This week’s headlines contain the all-too-familiar claim that a new study shows that vaping does not aid in smoking cessation, or even hinders it. Once again, this interpretation of the study results is wrong, and once again it is for most of the same reasons similar past claims were wrong. The error is not complicated, but it is easy for tobacco controllers to use it to manufacture anti-vaping results: Not every smoker who vapes is trying to switch, and smokers who have been vaping a while have already demonstrated they are not likely to switch. An additional common error is ignoring the fact that smokers who use vaping to quit are often having a harder time quitting (the majority of quitters just walk away from smoking when they finally decide to). This too is a problem with the new study, though not the main one.
The new paper, “Are electronic nicotine delivery systems helping cigarette smokers quit? Evidence from a prospective cohort study of U.S. adult smokers, 2015–2016,” by Scott R. Weaver, Jidong Huang, Terry F. Pechacek, John Wesley Heath, David L. Ashley, and Michael P. Eriksen appeared in the journal PLOS One. The authors are part of the FDA’s pet research shop at Georgia State University, and while they did not spin their results as hard as some of those reporting them did, they definitely did interpret their data in way that appears designed to support the FDA’s anti-vaping plans.
The study followed a cohort of smokers and found that those who also vaped at baseline (in 2015) were considerably less likely to have been smoking-abstinent for 30 days at follow-up (in 2016). (This was referred to as “quitting” smoking, though most smokers who are abstinent for 30 days actually start smoking again.) It turns out that almost 10% percent of the vapers became smoking-abstinent, which is actually quite impressive. Any intervention that got 10% of a random sample of adult smokers to quit in a year would be considered wildly successful. The “considerably less” result was driven by the fact that an astonishing portion of the non-vaping smokers, almost 20 percent, were smoking-abstinent at follow-up.
The authors acknowledge that this was odd, but they do not investigate it. Proper scientists, upon encountering something this strange in their data, would endeavor to explain it. If they could not, they would be doubtful about all their results. In the dysfunctional world of public health research, however, it is considered adequate to simply note that something might have been a problem and then just ignore it in the analysis. Thus, it could be that there was something unusual about the non-vapers in the sample, while the vapers were more typical, and that alone created the main result. But even setting that aside (i.e., assuming — for no good reason — that whatever produced a sample of non-vaping smokers who were unusually likely to quit soon also affected the sample who vaped), there is still the generic problem with studies like this.
A previous science lesson addressed the difference between stock and flow, the importance of which is emphasized in the (few) good epidemiology textbooks and courses, but which is widely ignored in the literature. As a general rule, if you take a random sample of people and ask “do you do X right now,” (measuring the current stock, also known as prevalence) most of those answering “yes” will be those who have done X for a long while and will continue to do so. This is true even when there are far more people who do X briefly. The stock at a given moment is mostly long-term Xers, and thus is a poor measure of the flow (also known as incidence) of everyone who initiates X. So, for example, at any given moment, most of the people with a headache are the minority who suffer from chronic headaches, even though over the course of a year almost everyone gets a headache. Similarly, most people who are in London live there and will continue to do so, but most people who have been there in the last year have already left. If you want to study how much getting a headache decreases food intake for the day, you do not want to collect a sample of people who have a headache right now, because most of them have a headache most days.
In vaping studies, if you gather a cross-section of smokers who currently also vape, most of them will be those who have been doing both for a while and, for the same reasons, will continue to do both. They will include dedicated smokers who vape some (to cut down or to deal with place bans) as well as smokers who hoped to switch entirely and failed to do so, but kept vaping some (they may still be hoping to switch entirely, but have already demonstrated it is less likely to happen than for those who succeeded). At the moment of sample selection, there will be a few “vaping success story” candidates — smokers who have recently started vaping and will soon switch entirely — but very few of them. They do not stay part of the stock very long. Thus, sampling people who have both vaped and smoked recently is a way to avoid including almost everyone who is a good candidate for switching. But the game is to pretend that it is a measure of whether vaping helps people quit smoking.
The authors use some phrasing that a careful reader could charitably interpret as saying, “merely being a vaper does not mean that someone is will soon quit smoking,” which is a valid interpretation of their results. But a few bits of subtle language is not good enough. The authors undoubtedly knew that their results would be interpreted as saying, “taking up vaping for the purpose of quitting smoking is useless or counterproductive,” and indeed some of their language basically says that. This is an obviously invalid interpretation.
It turns out that buried in the tables is some useful information about the use of vaping to quit smoking. For subjects who made a self-defined quit attempt during the study period, those who did so by trying to switch to vaping were smoking-abstinent at follow-up (“had quit” in the language of the paper) more than 20 percent of the time. This is quite impressive given that, as explained above, the population sampling method excludes almost all smokers who are inclined to switch to vaping quickly after trying it. This contrasts with only a few percent who became abstinent when their quit attempt included NRT or other “approved” methods.
Also buried in the results is the fact that subjects who started vaping during the study period were much more likely to have become abstinent at follow-up than those who vaped at baseline (though still at a rate lower than the unexplained strangely high rate for the non-vapers). This is consistent with the above observations: Some smokers tried vaping for the first time sometime that year and quickly quit smoking because of it. By contrast, for most of those who were already vaping at baseline, that possibility was already in their past; they were still smoking and thus in the study only because switching had already not worked for them. It is also interesting that more than a third of the vapers (at baseline) who became smoking abstinent also gave up vaping. This suggests that they were among those who vaped only because it was part of their smoking habit, not because they wanted to use it to replace smoking.
The authors carefully avoid acknowledging that smokers who vape are not all the same, let alone that their sampling method selected those who were least likely to switch. They report some statistics from a question about using vapor products to quit smoking, but it is such an uninterpretable mess one can only assume that it was included in the survey to create the illusion of measuring inclination to switch. They did not simply ask “do you want to switch to just vaping?” and “did you start vaping to try to quit smoking?” (let alone “if so, why haven’t you switched?” and “…do you still plan to switch?”). Instead the survey asked about the “importance” of quitting smoking as a motivation for vaping, and the possible answers were an arbitrary scale. Again, this seems designed to avoid getting legitimate measures of the effects of vaping in order to quit smoking.
An interesting side-note is that the authors did include questions about what types of products and flavors the vapers used. They attribute this to following the guidelines from a recent paper from David Abrams’s shop, in which the authors propose “quality standards” for research on vaping-based smoking cessation, which include looking at such variables. This illustrates the danger of providing rules-of-thumb in public health research. General recommendations quickly get turned into fetishes by the poor-quality scientists who populate the field. Major examples include any use of the phrase “causal criteria” and the myth that there is a hierarchy of what is always the better study design study design. Indeed, the present case is an example of a cohort study, where a population is followed over time, being far less useful than a decent retrospective or cross-sectional study, though those mythical hierarchies always say cohort studies are better.
The Georgia authors noted that the “standard” says they should look at whether open-vs-closed systems or choice of flavors affected the results, so they did. They then patted themselves on the back for doing everything right, never mind that the fundamental flaws in the study design rendered such comparisons just as meaningless as the main result. In fairness to the Abrams group, their “standards” list also includes statements that amount to “do proper epidemiology in the first place.” But these are necessarily vague and so will be ignored — in favor of the simple rules that can be fetishized — by anyone who does not already understand how to do proper sample selection and such (or is intentionally doing it wrong).
The new study and others suggest that long-term vaping alongside smoking does not make smoking cessation more likely, which is useful to know. One implication is that targeted efforts to get such smokers to just go ahead and switch might increase smoking cessation. It would also be useful to know if it is genuinely the case that long-term vaping actually decreases the probability of smoking cessation, but we are not going to learn that from the propaganda-generating studies we have. (There is the complication that smokers who persist in vaping are probably less “able” to just quit than average, which these studies effectively pretend is not the case.) What we will definitely never learn from studies that prospectively follows an existing stock of vapers, however, is anything about whether trying vaping for the purpose of quitting smoking is useful.