A year-old fatally-flawed student project, which has been misinterpreted as showing vaping causes heart attack risk, has recently been resurrected by — who else? — Stanton Glantz. It was originally presented as a research day poster (a standard exercise to teach students how to go through the motions of doing and presenting research) by students not affiliated with Glantz. Somehow he cajoled, bribed or otherwise arranged to have himself named as an author of their work and recently presented a new version in a poster at the Society for Research on Nicotine and Tobacco conference.The problem is that there is no possible way the data they used can be used to estimate risks from vaping due to the unmeasured confounding effects of smoking.
Greg Conley took a photograph of the poster:
And here’s the Glantz poster pic.twitter.com/gtzK9j8EIQ
— Gregory Conley (@GregTHR) February 24, 2018
The analysis used data from the National Health Interview Survey, a blunt instrument that contains information that is a mile wide but an inch deep. It simply compared the history of heart attacks (acute myocardial infarctions or MIs) among vapers to non-vapers, controlling for age, for an extremely crude measure of smoking status (current, former, never), and for a few other arbitrarily-selected variables that the researchers threw in because they happened to have them. The original student poster reported a 43 percent higher rate of MIs among the vapers but appropriately concluded only that the possible association was worth investigating. Glantz apparently introduced them to model shopping techniques and the career benefits of overstating one’s results. The version he presented managed to double the estimated risk elevation, and he declared that this shows that vaping causes MIs. Glantz had his university blast out a press release that managed to even further exaggerate the already faulty results.
Previous criticisms of this analysis have fixated on relatively unimportant concerns, as well as out-and-out red herrings. It was originally a student research day project, but this does not represent appreciably less quality control than the typical article published in a public health journal. It has not yet appeared in a peer-reviewed journal, but it will, and the peer-review process will do nothing to correct the errors noted here.
The one substantive concern that most critics have noted is that the data does not identify when the MIs took place. Many, probably most, of the MIs occurred before someone started vaping. Obviously it is absurd to blame vaping for an MI that occurred before someone vaped. This is a sufficient problem that no honest researcher would attempt to draw firm conclusions from these results. However, the effect of this problem may be quite small. A roughly equal prevalence of old MIs will have also been included for the non-vapers. That adds a lot of noise, but it does not necessarily bias the results. A smoker who has an MI is more likely to quit, and thus perhaps more likely to take up vaping than average, which would bias the result. However, this bias is presumably mitigated by smokers who have MIs and are motivated to quit unaided.
It is a mistake to fixate on this simple limitation rather than the core problem, because doing so basically endorses similar bad science that will appear in the future. Another dataset or a future NHIS wave could eliminate this problem collecting information about the timing of past MIs. (Some commentators have claimed that a cross-sectional study like NHIS could never determine when something happened. It is difficult to understand how someone could think this — after all, such surveys always collect data on when someone was born. It just happens that NHIS, as a very shallow dataset, did not collect this.) Another study could look at an outcome measure where timing is not an issue. Either of these would easily address what some commentators are implying is the main problem, but they would not eliminate the actual main problem.
This research approach is fatally flawed because of unrecorded details of someone’s smoking history. Glantz has absurdly insisted that the study controlled for smoking by including variables for current and former smoking. But smoking is not a dichotomous (or trichotomous) exposure. Smokers who smoke more, or used to smoke more, have higher risk of disease. Former smokers who quit recently are at much higher risk than those who quit long ago. Former smokers who vape will have quit smoking relatively recently and are more likely to have been hardcore smokers (less intense smokers who could take-it-or-leave-it are more likely to just stop, not switch to vaping). Current smokers who vape are probably more intense smokers who are motivated to search for a useful aid to quitting or are more interested in a substitute for when they cannot smoke.
Decades of research on smokeless tobacco have made clear that attempting to “control for” smoking is hopeless. Contrast the recently reported study about cigar and pipe smoking: It had serious flaws, but at least the researchers knew enough to look only at exclusive cigar and pipe users, rather than to try to control for cigarette smoking among those who used multiple products.
Smoking doubles the age-specific risk of MI. A realistic hypothesis about risks from vaping would be that it increases the risk by something in the range of five or ten percent. This means that the effects of past or current smoking would have to be measured and controlled for extremely precisely, or else the residual confounding error they create will overwhelm the effect of interest. It is like trying to measure how much air conditioners raise the outside air temperature in a city. Weather has a much greater effect, and a minor change in the weather will swamp any effect of the air conditioners. Moreover, air conditioners run more when the weather is hot. If you do not have an extremely good measure of the day’s weather effects, there is no way to control for it and thus the results are worthless.
The best way to deal with this would be to look only at exclusive vapers (never-smokers), but there are almost no exclusive vapers, at least not old enough to include in disease outcome studies. Thus any truth-seeking effort to assess health impacts of vaping would have to do an extremely good job of measuring for intensity of smoking and recency of quitting, and control for those. But epidemiology research seldom involves trying to do things right; it is usually just a matter of using whatever data is convenient, even if it is not possibly sufficient to answer the question (more about this in the next science lesson article). When the data only measures smoking in terms of current-former-never, it is literally useless for assessing any health effects of vaping.
A running Twitter conversation about the Glantz poster that included this reporter, a graduate student who was running analyses of the NHIS data and others produced some further observations (interested readers can find its branches downstream from this tweet). There was a prediction that the next “discovery” using this same data and method will be that vaping causes asthma. It will again suffer from the same fatal flaw.
One amusing note from that conversation is about the data waves that were used. The original authors used 2014 NHIS data but the Glantz version added 2016 NHIS data, perhaps just because it had become newly available, or perhaps because it increased the results. But 2015 was mysteriously excluded. It turns out that adding in 2015 does not change the results, so it looks like it was omitted because the vaping data was stored in a different file that year, and Glantz et al. apparently did not find it. While this had no material impact, it speaks to the general sloppiness and lack of interest in truth-seeking of the authors.
It was observed in that conversation that it is plausible that vaping causes some modest MI risk, in spite of the fact that smokeless tobacco does not seem to. This is obviously quite different from saying there is evidence supporting the claim. But it does suggest that it would be worth investigating. Vapers and would-be vapers have far more interest in learning this than anyone, and should welcome legitimate attempts to find out information that allows them to make better informed choices. Unfortunately, all available resources are devoted to creating alarmist claims that provide no actual information.