More and more the latest health research is accessible online. In fact, it’s more than just accessible, it’s showing up in our news feeds and inboxes. And while that’s mostly a good thing, too often, in my opinion, news about scientific findings is delivered alongside a product, a supplement, or a drug (pharmaceutical) proclaiming to fix all things for those of us in perimenopause.
Exciting (or scary) eye-catching headlines about new research encourage us to click to “learn more”. This is particularly tempting if the headline offers to ease a symptom we are dealing with or touts a way to avoid something we are concerned about (e.g., dementia, breast cancer, heart disease).
Perimenopause with its fluctuating hormones can be enough to endure. We shouldn’t need to work hard to decipher perimenopause research findings that may ease our way, but sadly we do. Here are six things to help you be a more critical thinker about the research you might read.
1. What TYPE OF STUDY was used?
2. What is the SAMPLE SIZE? (i.e. how many people were studied?)
3. Has the study been REPLICATED?
4. WHO was studied? Are they a REPRESENTATIVE sample? What SPECIES?
5. What is the TIME HORIZON studied?
6. What is the FUNDING SOURCE? Could there be a CONFLICT OF INTEREST?
1. What TYPE OF STUDY was used?
All research starts with a question and a hypothesis (or theory) based on previous research about possible relationships that might answer that question. The different types of studies depend on whether their intent is to explore a hypothesis or to test a hypothesis.
If you take away only one thing from this post, know that there are 2 main kinds of studies — Observational and Experimental Studies.
Observational studies explore a hypothesis.
Experimental studies test a hypothesis.
Read on to learn about what you CAN and CANNOT conclude from each type of study.
OBSERVATIONAL STUDIES vs. EXPERIMENTS
To understand the two main types of studies, we will use the relationship between taking estrogen and heart disease as an example.
The type of the study determines what you can conclude from the findings. From an observational study, you can only conclude that there was or wasn’t a correlation (or association or relationship) between a variable and an outcome. Using the example of taking estrogen (a variable) and heart disease (an outcome), you can observe the relationship, but you cannot conclude that estrogen caused or prevented heart disease.
To establish that one variable (estrogen) led to an outcome (heart disease), you need to conduct an experiment. A double-blind randomized controlled trial produces the strongest evidence and eliminates bias. We’ll break down what that means below.
OBSERVATIONAL COHORT STUDIES
With an observational study, you are looking at what happens in two or more different groups of volunteers, who are living their normal lives. These groups are referred to as cohorts. You can do a cohort study in a forward-looking way (prospectively) or a backward-looking way (retrospectively).
Observational cohort studies: prospective
In a prospective observational study, you would recruit 2 groups of people: one group that has decided to take estrogen and a second group that does not take estrogen and you would “follow them” for 15 years by gathering data via surveys, possibly blood samples, scans or other kinds of testing at predetermined intervals, say once a year. At the end of the study, you would be able to establish that there was or wasn’t a correlation (or association or relationship) between taking estrogen and heart disease, but you would not be able to say estrogen caused or prevented heart disease.
Observational cohort studies: retrospective
To study the same question with a retrospective observational study, you would identify two groups of people. One group that had taken estrogen in the past and a second group that had not taken estrogen. The study would determine in which participants’ heart disease had developed, and when in relation to taking estrogen, by looking at past data from electronic medical records and surveying participants. Again, at the end of the study, you would be able to say that there was or wasn’t a correlation (or association or relationship) between taking estrogen and heart disease, but you would not be able to say estrogen caused or prevented heart disease.
Observational cohort studies: cross-sectional studies
Here you ask many people a series of questions, conduct tests, do scans or collect blood samples at a single point in time. A survey is an example of a cross-sectional analysis.
EXPERIMENTS
Experiments: Randomized Controlled Trials
In order to establish cause and effect, you need to conduct an experiment, a randomized controlled trial (RCT). In this type of trial, you introduce an intervention (estrogen) and study its impact on your desired outcome: getting or preventing heart disease.
There are four key elements of RCTs:
- At least two groups (aka conditions, arms) including a “control” condition (sometimes referred to as placebo or inactive) and one or more “treatment” condition(s) (sometimes called active) to which participants are randomly
- An intervention (in our example, estrogen: something the investigator is introducing to the treatment group. *Note: the control group gets something that looks just like the treatment but is inactive called a placebo.
- Pre and post-measurements of the disease outcome (in our example heart disease)
- Blinding (sometimes called masking) of participants to the conditions of their study group, they don’t know whether they are in the treatment group or the control group. That is they don’t know which they are taking, the study drug or the placebo. In a “double-blind” trial, the study investigators also do not know which group is getting the treatment. *Note: only the statisticians who are receiving continuous data, can break the blind after a certain point in the study if there appears to be a problem that needs consideration of the advisory group, e.g., or stopping the study if there is harm occurring.
A double-blind randomized controlled trial eliminates the most bias and yields the strongest evidence.
Experiments: Non-Randomized Controlled Trials
Here again, two or more similar groups of people are studied. One group, the control group, gets a placebo. The other group or groups get the intervention (e.g., a drug, a behavioral program, an educational program). This type of trial should be viewed with caution. Subjects are not assigned randomly which allows for bias and results that do not produce as strong evidence as an RCT.
Experiments: Meta-analysis of RCTs
The strongest of all evidence in making health care decisions comes from a “meta-analysis”— a study that statistically combines the results of many RCTs. A meta-analysis can also be done on observational studies. This meta-analysis would be stronger evidence than a single observational study but not as strong as a meta-analysis of RCTs.
QUALITATIVE STUDIES
In addition, to these two main types of study, there are also Qualitative studies. These are used to characterize people’s attitudes, experiences or behaviors but not to explain the relationship between variables. They generate non-numerical data, sometimes quotes. These have been of great benefit to women’s health, an area where there is still so much that is not understood about our experiences on the path to menopause. In fact, the WLB research team has just published this qualitative study about experiences of the perimenopausal healthcare interaction. The data are quotes about healthcare interactions. (We think) a fascinating read!
Do observational studies and experiments (randomized controlled trials) always come to the same conclusions?
NO!
A well-known, real-life example is The Women’s Health Initiative (WHI). Since observational studies only tell you about the correlation (or association or relationship) between a variable and an outcome, you can’t be sure that a certain variable is causing the outcome. For example, from an observational study, you might see that those who took estrogen were less likely to develop heart disease. This is what was found in 4 observational studies in the 1990s1,2,3,4. From these observational studies, it was thought that estrogen was the reason for fewer cases of heart disease and that taking estrogen might prevent heart disease.
To test that idea, a randomized controlled trial was needed to determine whether there was, in fact, a cause-and-effect relationship between taking estrogen and less heart disease. And this was the goal of one of the four parts of the Women’s Health Initiative Study — to answer the question, does taking estrogen prevent heart disease?
It turned out that the answer was no, estrogen did not prevent heart disease. There must have been other factors that the observational studies didn’t identify or measure that led to lower heart disease. It is important to make sure that the two groups are similar at baseline. And in this case, they were not. The women choosing to take estrogen were more likely to have a regular doctor, tended to have more education, higher-paying jobs, less use of cigarettes, fewer with diabetes and healthier physical activity and diet. All of these factors can (and did) affect the outcome of heart disease.
In fact, in the WHI trial, participants taking hormones tended to have more heart disease than those who did not, and the trial was stopped early. Critics of this trial point out that many of the women participants were older, more than 10 years since their final menstrual period (the average age was 63). Estrogen therapy might have different effects on the risk for heart attack in older postmenopausal women. Subsequent subgroup analysis of the WHI data examined the results only in younger participants, 50-60 years old and within 10 years of their final menstrual period. Those results did not show the same increase in heart disease. Another RCT and estrogen and heart disease has not been done since.
The strongest of all evidence in making health care decisions comes from a “meta-analysis”— a study that statistically combines the results of many RCTs.
2. What is the SAMPLE SIZE? How many people were studied?
Bigger is usually better! So often I see a really exciting title of a research study only to find there were only 20 or 30 people studied. This is ok for a pilot study that aims is to test whether there might be a finding before spending more money on a bigger trial, but it’s hard to draw significant conclusions from small samples.
It’s up to the investigator to make sure there are enough participants to be able to detect a statistical difference between the treatment group and the control group if there is one. However, you can have a small RCT with a significant difference between those in the treatment versus the placebo group that demonstrates a very strong effect
When research can’t be replicated or there have been contradictory findings, you should read with skepticism. Be curious about why the findings are different. Why wasn’t the study able to be replicated?
3. Has the study been REPLICATED?
One of the tenets of a scientific study is that the findings should be possible to be repeated by another group of scientists. Replication increases confidence in the findings. This is why every study includes a detailed description of the methods so that another group of researchers can duplicate what they did and see if they find the same result. When research can’t be replicated or there have been contradictory findings, you should read with skepticism. Be curious about why the findings are different. Why wasn’t the study able to be replicated?
Often new results are presented for the first time at a conference, and they garner lots of excitement and press coverage. However, they need to be published in full after other scientists have reviewed them (called “peer review”). In addition, they should be replicated before being seen as “ground truth” or fully believed.
To this very point, a group at UC San Diego researched this very topic in May of 2021. Their paper’s title was, “Nonreplicable publications are cited more than replicable ones”.5 ScienceDirect’s coverage of this paper was titled, “A new replication crisis: Research that is less likely to be true is cited more”. Papers that cannot be replicated are cited 153 times more because their findings are interesting.”
This is why a meta-analysis that combines the data from many randomized controlled trials is the strongest evidence.
4. WHO was studied? Are they a REPRESENTATIVE sample?
It was only in 1991, that an Office of Women’s Health was established at the National Institute of Health. Prior to that, most pharmaceutical research was conducted on men. Those with menstrual cycles who could possibly bear children were considered too variable and risky to study. A possible intervention could harm a fetus, or the normal hormonal fluctuations of a menstrual cycle might influence the drug studied. But this means that much of what we know about medications is based on men and the theoretical 70 kg/ 150 lb male at that!
But this means that much of what we know about medications is based on men and the theoretical 70 kg/ 150 lb male at that!
As another example of representation, most of the participants in research about perimenopause have been White. Exceptions include The Seattle Midlife Women’s Study6 which engaged African American, Asian American, and White women, The Penn Ovarian Aging7 study which recruited a random population-based sample that was half Black and half White and the ongoing Study of Women’s Health Across the Nation8, begun in 1994, which explicitly recruited participant volunteers from four ethnic groups.
Also note, What SPECIES was studied?
I also see studies with eye-catching titles only to realize after reading further that the study wasn’t done on humans.
Early studies to test a concept can (and often need to) be done in cells, organelles, mice, rats, dogs, or monkeys. And each is a risk-reducing way of gathering data before human trials, but the applicability to humans should not be overstated and it often is.
5. What is the TIME HORIZON studied?
How long is the group observed (observational studies) or how long is the intervention studied (experiments)? Weeks, months, years? How often are data collected within that timeframe? The more data points and the longer the study, the stronger the data set.
6. What is the FUNDING SOURCE? Could there be a CONFLICT OF INTEREST?
Any conflicts of interest have to be disclosed in all research. It’s worth checking on the affiliations of the authors. Not infrequently studies of products are funded by the companies that make the product. Determine who has paid for the study. Those studies that are paid for by national or international granting bodies like the National Institutes of Health (NIH) are the most reliable since they will have had the strongest reviews and the least bias.
References
- Stampfer MJ , Colditz GA. Estrogen replacement therapy and coronary heart disease: a quantitative assessment of the epidemiologic evidence. Prev Med. 1991;20:47–63.
- Grodstein F , Manson JE , Colditz GA , Willett WC , Speizer FE , Stampfer MJ. A prospective, observational study of postmenopausal hormone therapy and primary prevention of cardiovascular disease. Ann Intern Med. 2000;133:933–941.
- Henderson BE, Paganini-Hill A, Ross RK. Decreased mortality in users of estrogen replacement therapy. Arch Intern Med. 1991;151:75–78.
- Grady D , Rubin SM , Petitti DB , et al. . Hormone therapy to prevent disease and prolong life in postmenopausal women. Ann Intern Med. 1992;117:1016–1037.
- Serra-Garcia, M., & Gneezy, U. (2021). Nonreplicable publications are cited more than replicable ones. Science Advances. https://doi.org/abd1705
- Freeman EW, Sammel MD. Methods in a longitudinal cohort study of late reproductive age women: the Penn Ovarian Aging Study (POAS). Womens Midlife Health. 2016 Jan 27;2:1. doi: 10.1186/s40695-016-0014-2. PMID: 30766699; PMCID: PMC6299955.
- Woods NF, Mitchell ES. The Seattle Midlife Women’s Health Study: a longitudinal prospective study of women during the menopausal transition and early postmenopause. Womens Midlife Health. 2016 Nov 9;2:6. doi: 10.1186/s40695-016-0019-x. PMID: 30766702; PMCID: PMC6299967.
- Avis NE, Crawford SL. SWAN: What It Is and What We Hope to Learn. Menopause Management. 2001;10(3):8-15. 161
