Studies in the behavioural and brain sciences reporting a major sex-dependent effect – that a drug, treatment or other intervention is more effective in one sex than another – are supported by appropriate evidence less than 25% of the time, an analysis finds.
The Proceedings of the National Academy of Sciences (PNAS) published the analysis of 200 recent articles with a claim of a sex- or gender-dependent effect in the title. The articles included studies on human and non-human subjects and spanned six brain-related research areas: behavioural sciences, clinical neurology, neurosciences, psychiatry, psychology and substance abuse.
“We found that studies in psychology had the highest rate of appropriate evidence – 39% of the published papers included statistical evidence to support the claim of a sex difference,” says Donna Maney, corresponding author of the study and professor of psychology at Emory University. “Research in neuroscience had the lowest rate of appropriate evidence, at just 18%.”
The low rate of appropriate evidence in neuroscience is particularly troubling, Maney says. She notes that claims of sex-dependent effects are more numerous in neuroscience, where such reports are currently being published at triple the rate seen in any other field.
“The high number of reports seen in neuroscience may be due to bias – neuroscientists looking harder for sex differences than other scientists,” Maney says. “But most current evidence shows that the brain is one of the least sexually differentiated organs in the body.”
Maney’s team was particularly alarmed by the number of calls for changes in clinical approaches that were based on faulty analyses. Many of the 200 articles they reviewed, for example, called for sex-specific approaches to suicide prevention, stress-related psychiatric disorders, substance-use disorder and psychopathy – all without providing statistical comparisons of effects across sex.
First author of the PNAS paper is Madeline Olivier, who did the work as an Emory student and has since graduated with a BS in psychology.
Summary of findings
- In 24% of the 200 papers, the effect compared statistically across sex and the results supported the claim of a sex-dependent effect.
- In 9%, the researchers tested for a sex difference, but the results were missing.
- In 9.5%, the sex difference in the effect was reported as not statistically significant, which was incompatible with the claim in the title.
- In 57.5%, the sexes were not statistically compared — the researchers did not test the claim in the title at all.
A logical error
Maney is a neuroscientist who studies hormonal and genetic influences on behaviour. For more than a decade, she has also focused on investigating how sex differences are tested for and reported in biomedical research.
One issue she emphasises is that, instead of comparing the sexes directly with each other, researchers often test for the effect in each sex separately. Although it might make sense on the surface, the practice reduces the number of subjects to the point where a real effect can be missed. If the effect is detected in one sex but missed in the other, researchers are vulnerable to a logical error: that the effect differs between the sexes, when they have not been directly compared.
To show that the sexes differ, females and males must be directly compared with each other in a statistical test. Most of the articles analysed by Maney and colleagues for the current PNAS paper did not do that. Instead, the researchers relied on the individual, within-sex tests – an invalid way of comparing the sexes that produces the illusion of a difference up to 50% of the time. “It’s no better than flipping a coin,” Maney says.
It’s also easy to miss true sex differences with a subgroup approach. For example, men and women could respond differently to a treatment but when the sample is divided in half and tested separately, the effect could be missed in both.
Maney cites the classic example of a large clinical trial showing that aspirin significantly reduced mortality from heart attacks. To illustrate the problem with the subgroup error, cardiologist Peter Sleight reanalysed the data by dividing participants into subgroups according to their astrological signs. Once the trial was split into 12 zodiac groups, the benefit of aspirin was no longer statistically detectable among the Libras and Geminis.
Sleight’s “findings” demonstrated how dividing a large group into subgroups can make a real effect disappear in some of the groups, even when the treatment is clearly beneficial.
“This problem is not new,” says Maney. “I made the error myself until I learned about it. “Many researchers don’t receive training in how to test whether an effect differs between two groups.”
A simple solution
To provide evidence that an effect differs by sex, the effect must be statistically compared between males and females, Maney emphasises. Only that approach can show sufficient evidence for a sex difference.
She designed an open-source tool, housed on the web at sexdifference.org, to help guide researchers to verify sex-specific effects.
Maney’s interests extend beyond statistical sex comparisons.
“Ultimately,” she says, “I would like to see researchers not treat sex as the most important variable in a biomedical study. Variation in participants’ weights, ages or habits, for example, likely explains variation in the effect of a treatment better than which sex category they are in.”
Original written by Carol Clark
Source: Emory University
