- A widely cited 2024 meta-analysis by Glantz et al misclassified diseases, double-counted cases, and relied on weaker study designs, experts say.
- Risk estimates based on inconsistent data may overstate the harms of vaping.
- High-quality studies find no evidence of serious respiratory harm in never-smokers who vape.
- Flawed evidence can mislead policymakers and discourage smokers from switching to safer alternatives.
Poor-quality research on vaping is distorting public health policy and discouraging smokers from switching to safer alternatives, a new editorial has warned.
Published in Internal and Emergency Medicine, the expert article critiques a 2024 meta-analysis by Glantz and colleagues, claiming it suffers from “major methodological flaws” that make its conclusions unreliable.
The Glantz study, published in NEJM Evidence, suggested that vaping could pose disease risks similar to smoking, especially for cardiovascular conditions. But the new review, by Rodu and colleagues, argues that those claims are based on shaky evidence.
“The credibility of any meta-analysis is directly dependent on the quality, comparability, and methodological rigour of the studies it includes,” they write.
The editorial says Glantz’s team grouped very different illnesses under broad disease labels. For example, erectile dysfunction and heart attacks were both classified as “cardiovascular disease.” Similarly, respiratory conditions like flu and chronic obstructive pulmonary disease (COPD) were lumped together “despite having distinct clinical profiles.”
Rodu and colleagues are also concerned that the Glantz paper relied too much on cross-sectional studies, with 76 per cent of its odds ratios coming from this type of research. But these studies measure exposure and health outcomes at the same time, so they can’t prove cause and effect.
“Cross-sectional designs assess both exposure and outcome at a single point in time… the evidence cannot support causal inferences,” the authors explain.
Many of the studies used in the Glantz meta-analysis also didn’t include basic timing data such as when participants started smoking or developed illness. This makes it “impossible to establish whether the exposure could plausibly have contributed to the health outcome,” according to the review.
The authors warn that repeating these types of errors on a large scale is undermining trust in public health science. They say: “The persistent repetition of such methodological shortcomings has now reached a scale that risks undermining the credibility of public health science itself.”
Double counting
Another issue is double-counting. Some included studies drew on the same national surveys (like NHIS or BRFSS), meaning some participants may have been counted more than once.
This could “artificially enhance the perceived consistency or precision of the findings.” Even statistical adjustments, such as inflating error margins, don’t fix the underlying problem, the review says.
The absence of cumulative exposure data is another key gap. Without knowing how much or how long someone has vaped, researchers can’t accurately assess risk. “This can obscure true dose-response relationships and conflate light, short-term use with heavy, chronic use,” the editorial notes.
Flaws in longer term studies
Even longer-term studies, which are often considered more reliable, had flaws. Many didn’t track changes in smoking or vaping habits over time. According to the review, this “undermines the validity of the conclusions drawn from such data.”
One exception praised by Rodu’s team is the Berlowitz study, which accounted for these behavioural changes and found “no statistically significant association between e-cigarette use and cardiovascular outcomes.” This finding “directly challenges the overarching conclusions of Glantz et al.’s meta-analysis,” the authors say.
Another example is the Xie study, which linked vaping to COPD. Upon closer inspection, Rodu et al. found that “nearly all of the individuals diagnosed with COPD… were either current or former smokers. In fact, only one participant with COPD had never smoked.”
Strong studies show no respiratory issues from vaping
This aligns with recent reviews of high-quality studies. One umbrella review looked at 12 systematic reviews and found “no convincing evidence of short- or medium-term respiratory harm associated with e-cigarette use.” Among never-smokers, “both reviews found no evidence of serious or sustained respiratory harm attributable to vaping.”
The Glantz analysis also failed to separate dual users (people who smoke and vape) from exclusive vapers or smokers. “Most analyses do not disaggregate these patterns, undermining meaningful interpretation of dual-use risk,” Rodu and colleagues note.
They also point to statistical mistakes, including incorrect use of the Bonferroni correction – a method for controlling false positives. The paper says it was applied “without an adequate explanation of which comparisons it was meant to control for or why such a stringent adjustment was appropriate.” This not only reduced the study’s power but also wrongly assumed that all comparisons were independent.
Another serious error was treating a non-significant result as proof that different study types are equivalent. “Mistaking a lack of statistical significance for evidence of equivalence is a common but serious error,” they warn.
‘Distorting public understanding’
These mistakes aren’t just academic. “The misuse of statistical tools in this context does not reflect a harmless misunderstanding,” the editorial says. “Rather, it represents a concern in scientific reasoning that can mislead readers, distort public understanding of risk, and skew the policy debate around e-cigarettes.”
This matters because such distorted findings can make people wrongly believe that vaping is as dangerous as smoking. That could stop smokers from switching to a less harmful option. The authors note: “Subjective evidence indicates that e-cigarette users consistently rate their health more favorably than smokers,” echoing guidance from UK bodies like the Royal College of Physicians and Public Health England.
“The consequences of flawed meta-analyses are not confined to academic debate,” the editorial warns. “Once established, these citations can take on a life of their own, being treated as definitive summaries of the evidence even when their underlying data and interpretations are deeply problematic.”
A call for better standards
The authors call for better standards in how meta-analyses are done and interpreted, especially in sensitive areas like tobacco harm reduction. “The strength of scientific conclusions depends not on the volume of data but on the integrity of the methods used to analyze and interpret it,” they say.
They conclude that meta-analyses must be done with care, saying: “Aggregating flawed or incomparable studies does not create strength through numbers. Instead, it creates the illusion of authority while concealing the structural weaknesses of the underlying evidence.”
