Why Was The National Academies Report So Wrong About The Gateway Effect?
As previously reported, the recent National Academies of Sciences, Engineering, and Medicine (NAS) report on vaping and vapor products declared that there is an evidence of a gateway effect – that vaping causes teenagers and young adults to start smoking. The reality is that there is no such evidence. So how did the report authors make that mistake? The easiest explanation is that they just lied about the evidence, as tobacco controllers often do. But in this case the problem seems to be genuine ignorance about how to do proper science.
Observational epidemiology and related social sciences are much more difficult to do well than clinical trials or most lab science. However, going through the motions is easy, and researchers can get away with sloppy work. As a result, public health attracts, retains, and promotes people who simply do not have the skills to properly conduct or interpret the core science of the field. In particular, they do not understand how to deal with confounding, which is the problem in this case.
Confounding exists when exposed individuals (vapers) have a different risk for the outcome (smoking) that is not caused by the exposure (i.e., they would be more likely than average to start smoking even if they never tried vaping, or if e-cigarettes never even existed). This is obviously the case for vaping and smoking. Some young people are quite averse to trying either, while others are positively inclined toward them.
My paper about vaping gateway claims explains the difficulty in detail. It is a serious scientific challenge to separate any gateway effect — vaping actually causing smoking — from the confounding. Indeed, it is basically impossible to sort this out using the methods employed by the existing studies. The authors of the NAS report apparently do not understand this. Most people doing epidemiology do not understand it. (The NAS report references my paper, but suggests the authors did not actually read it, let alone understand it.)
The NAS authors spend six pages basically just saying that someone’s vaping can either cause later smoking, prevent it, or neither. They spend two pages on the importance of vaping preceding smoking rather than the other way around — a useful but trivial point. They spend about one page of hand-waving examining the huge challenge of confounding.
Their poor understanding of confounding is nicely summed up in their “framework” that is their basis for deciding the association was causal rather than confounding. They claim that this conclusion is supported by:
(1) Strength of the association; (2) consistency across studies, investigators, individuals, research methods, replications; (3) temporal precedence of e-cigarette use relative to combustible tobacco cigarette smoking; (4) comprehensiveness by which potential confounding effects were addressed and ruled out by covariate adjustment or other methods; and (5) dose responsivity in the association
Point 3 is valid. But 1, 2, and 5 are dead wrong, offering no distinction between causation and confounding at all. A major confounding problem, as exists in the present case, will be consistent across studies and produce a strong association. The dose-response for the confounding can easily be greater than for the effect of interest (i.e., someone with a stronger propensity of smoke is likely to also vape more).
Point 4 sounds sciency, but it further demonstrates the authors’ simplistic understanding of confounding. They appear to have no idea what would constitute comprehensiveness, nor what methods could be used other than covariate adjustment. (I offer some examples in my paper, but the NAS authors appear unaware of them.) Most important, the idea that confounding could be “ruled out” in this case is absurd. It is difficult to convey how laughable this is. The confounding exists. It still exists after it is partially “controlled for.” The best we could ever hope for it to try to detect some hint of gateway causation in spite of it.
Using “control variables” — other data used to try to statistically remove confounding effects — is more complicated than standard epidemiological practice suggests. Standard practice consists of taking whatever variables happen to be available and hoping, based on nothing, that they control for whatever confounding might exist. Good practice requires identifying causal pathways that create the confounding in advance, and then collecting data that is optimal for dealing with them (and, if that is not possible, being ready to concede the confounding cannot be controlled for). The NAS authors hint at a basic understanding of this by trying to list confounding pathways, which they identify as: “risk-taking” propensity, access and exposure to products, and “sociodemographic factors.”
To deal with “risk-taking” propensity (by which they really mean rule-defying and sensation-seeking), researchers should measure variables that reflect this tendency but are independent of tobacco use. They should try to collect data on other drug use, truancy, violating traffic laws, and such. In theory, these could offer a perfect measure of “risk-taking” propensity, making it possible to see if vapers were more likely to smoke later when compared to non-vapers with the same propensity.
But a hint of understanding is as far as it goes. There are three major problems here, each a fatal flaw in the NAS analysis. First, though the NAS authors correctly suggest the need to identify confounding pathways and use that assessment to determine optimal control variables, the authors of the actual studies did not do this. They engaged in the standard practice of just throwing in whatever variables they happened to have, and unrealistically assuming that this controlled for confounding. Indeed, “sociodemographic factors” is really a way of trying to dress up that approach as having some scientific basis. Is there any reason to believe that variables for ethnicity or geographic region will reduce the confounding, rather than increasing it or just introducing noise? No. But the data is easy to collect, so they throw it in.
Second, the story about perfectly measuring risk-taking propensity is an unrealistic simplifying example. Researchers can only hope for a pretty good measure of propensity (and in this case, did not even have that). This means the control variables can, at best, eliminate part of the confounding. So when imperfect control variables seem to reduce confounding, the proper conclusion is that there is almost certainly further “residual confounding” in the result. Thinking, “throwing in these variables changed our results and so therefore now we have no more confounding,” as the NAS authors and the original study authors did, is both common and absurd. The proper conclusion is that if there is any gateway effect, it remains impossible to sort out from the substantial residual confounding.
Third, that list of pathways leaves out the most important source of confounding: whether or not someone likes consuming nicotine. One-fifth to one-third of the population really likes consuming nicotine, and they do so even when the state tries to force them to stop. A slightly larger portion of the population dislikes consuming nicotine and will avoid it even when it is socially acceptable and low-cost. In between is a distribution of positive but less intense appreciation. This alone creates a huge association (confounding) between vaping and smoking behavior. There are methods that could be used to tease this out, but the authors of the studies and the NAS report seem unaware of them. At best, some studies have a measure of parents’ smoking status, which is an extremely rough proxy for whether someone likes nicotine. Thus, even apart from the weakness of what was done to address confounding, the authors did not even recognize the main source of confounding.
It is informative to consider what these same methods would show if applied to other data. Is eating at McDonalds a gateway to eating at Burger King? Among people who have never eaten at BK, those who currently eat at McDonalds are undoubtedly more likely than average to do so. They like burgers, they are willing and able to eat fast food, and so on. Perhaps there are a few who visit BK only because of their history with McDonalds (actual gateway cases), but we cannot know because many would have done so even if McDonalds had never been founded. Indeed, in a world without McDonalds, they would probably be more likely to already frequent BK.
To take an even clearer example, is smoking a Marlboro from manufacturing batch #123456 a gateway to later smoking a Marlboro from batch #123500? Assume the data shows that within a population, the former is strongly associated with the latter. Is it causal? Of course not. Both batches were shipped to where particular Marlboro smokers shop, so they smoked from one and then the other. If the first had been shipped elsewhere, they would still have smoked from the second. The strong association is obviously not causation. Now imagine that the researchers tried to control for the usual variables; this would adjust away a lot of the association, but not nearly all of it. The same methods that the NAS authors think demonstrate a gateway effect would still conclude that this, as well as the fast food example and countless other non-gateway examples, was shown to be causal.
In short, all of the observed associations are easily explained by confounding, and the NAS authors’ claims to the contrary demonstrate their lack of understanding about confounding.
This is not an bold claim. The typical public health researcher does not understand any of this. They do not understand that confounding can create a strong and consistent association that shows a dose response. They do not understand that blindly throwing variables in to the statistics is not a useful way to control for confounding, no matter how many such variables are in a dataset. They do not understand that imperfect measures of a confounding pathway result in incomplete correction for confounding. And, despite being people themselves, they do not seem to understand that people make consumption choices mostly based on what they enjoy consuming.
Follow Dr. Phillips on Twitter