As The Washington Post first reported, Jay Bhattacharya, the acting CDC director, recently blocked a routine study on COVID-19 vaccine effectiveness from publication in the agency’s Morbidity and Mortality Weekly Report. The study found that this season’s vaccines reduced hospitalizations among healthy adults by roughly half. His objection was methodological. Now, Science reports that at an April workshop convened by the National Academies of Sciences, Engineering, and Medicine, Bhattacharya called the study’s methodology “crap” and “logistically ridiculous,” adding: “Any econometrician that you show this method to will go, are you freaking kidding me?”
The method is called the test-negative design, and it has been used to estimate vaccine effectiveness against influenza and other respiratory viruses for two decades.
The statistical questions Bhattacharya raises are not frivolous. But the scientific literature on this method is far more developed than his remarks suggest.
How the Test-Negative Design Works
The test-negative design is a variant of the case-control study. In the case of the COVID-19 study, for instance, patients presenting at clinics with acute respiratory illness are all tested for the pathogen of interest, typically by PCR. Those who test positive are classified as cases. Those who test negative become controls. Vaccine effectiveness is estimated by comparing vaccination rates between the two groups.
The key statistic is the odds ratio. Consider a simple example. Among 100 patients who test positive for influenza, suppose 20 are vaccinated and 80 are not. Among 100 who test negative, suppose 50 are vaccinated and 50 are not. The odds of vaccination among the positives are 20/80 = 0.25. The odds among the negatives are 50/50 = 1.0. The odds ratio is 0.25/1.0 = 0.25, and vaccine effectiveness is estimated as 1 minus the odds ratio, or 75%. As epidemiologists Natalie Dean and Avnika Amin explained in JAMA, in practice this is done with logistic regression, adjusting for confounders like age, calendar time, comorbidities and geographic location.
The design exists to solve a well-known problem. People who choose to get vaccinated tend to be healthier and more engaged with the healthcare system. In a standard study comparing vaccinated and unvaccinated populations, this “healthy vaccinee” effect can make vaccines look better than they are. The test-negative design addresses this by restricting the comparison to people who all presented with illness and got tested. Both cases and controls are sick; they differ only in whether their illness is caused by the target pathogen. Healthcare-seeking behavior, the major confounder in observational vaccine research, is controlled at enrollment.
The design emerged for influenza surveillance in the mid-2000s and was adopted worldwide during the COVID-19 pandemic.
The Statistical Objections
The biostatistics literature has been candid about the design’s limitations for more than a decade.
The most discussed concern is selection bias. Sheena Sullivan and colleagues formalized the problem in a 2016 American Journal of Epidemiology paper using directed acyclic graphs, the standard tool of modern causal inference (and a favorite among economists such as Bhattacharya). Their analysis showed that conditioning on healthcare-seeking, the very feature that is supposed to make the design work, can introduce what is called “collider stratification bias.” The idea is this: both vaccination and infection can independently cause a person to show up at a clinic. When you restrict your study to people who showed up, you are conditioning on a variable that is a shared consequence of both the exposure (vaccination) and the outcome (infection). In statistical terms, clinic attendance is a “collider,” and conditioning on it can create a spurious association between vaccination and test result even if none exists in the broader population. A companion invited commentary by Westreich and Hudgens titled “Beware the Test-Negative Design” sharpened the point: there is no formal guarantee that conditioning on testing eliminates the bias it is supposed to remove.
A second concern is more fundamental. Joseph Lewnard, Christine Tedijanto, Ben Cowling and Marc Lipsitch derived the mathematical relationship between the test-negative odds ratio and true vaccine effectiveness. They showed that the odds ratio recovers the vaccine direct effect only when two conditions hold: vaccination decisions are uncorrelated with exposure or susceptibility to infection, and the vaccine confers “all-or-nothing” protection, either blocking infection entirely or doing nothing. Most real vaccines provide partial, “leaky” protection. When they do, the odds ratio can be biased, and that bias can create misleading patterns in the data, including apparent waning of effectiveness that may not reflect true biological waning.
A third problem is severity attenuation. If a vaccine makes breakthrough infections milder, vaccinated individuals may never feel sick enough to visit a clinic. Their absence from the sample biases the estimate.
These concerns bear directly on how vaccine effectiveness is estimated for the pathogens that matter most. Because of this, Dean and colleagues have published methodological explainers in both The New England Journal of Medicine and JAMA, acknowledging the limitations while describing why the design remains the best available tool for real-time vaccine monitoring.
Validation Against Randomized Trials
Given these objections, how do we know the design actually works? The strongest evidence comes from direct comparison with randomized, placebo-controlled trials.
In 2013, Gaston De Serres and colleagues reanalyzed data from four RCTs of live attenuated influenza vaccine, re-estimating vaccine effectiveness using the test-negative design applied to the same trial data. Their estimates were “virtually identical” to the per-protocol analyses. The core assumption, that the vaccine does not affect the risk of non-target respiratory illnesses, held in every dataset.
Leah Andrews and colleagues recently extended this to COVID-19 by constructing test-negative datasets from five phase 3 vaccine trials across 16 countries. The concordance correlation was 0.86. They also tested a key assumption: that the controls, people who tested negative, look statistically the same whether they received the vaccine or the placebo. If the vaccine were somehow changing who shows up as a control, the design would be compromised. It was not. The median vaccine effectiveness against non-COVID illness was just 7.7%, with most confidence intervals crossing zero, meaning the vaccine had no meaningful effect on non-target diseases, exactly as the design requires.
None of this means the theoretical biases are absent. It means that in practice, at least for influenza and the COVID-19 vaccines tested so far, those biases are small enough that the design recovers estimates close to those of randomized trials.
What Is Actually at Stake
The test-negative design was built to solve a specific problem: monitoring vaccine effectiveness in real time, across seasons of antigenic drift and viral evolution, using existing healthcare infrastructure. The validation studies suggest it works for both influenza and COVID-19, though the assumptions require ongoing scrutiny as pathogens and populations change.
The researchers who have been most critical of the test-negative design – scientists like Westreich, Lewnard, Sullivan, Cowling, and Lipsitch – are also the ones doing the hardest work to improve it. Their critiques appear in the same journals as their proposed solutions. The test-negative design has known biases, known assumptions and a track record of producing estimates that hold up against randomized trials. That is a strong foundation for monitoring whether vaccines work. The question going forward is whether the people making decisions about public health are willing to build on it.
