I began writing this series of posts as a way of systematizing a number of ideas—some of them mere discomforts—that I’ve been accumulating as I’ve learned to to do technical/statistical/data-driven work in COVID-19 epidemiology over the course of the past year or so. I don’t exactly know what to do with these ideas: not being a trained life scientist, I don’t have a great track record at writing life science papers, and in any event I’m not certain that the things that I’ve discussed in the past few posts could be fit comfortably in a “real” scientific paper. There isn’t a lot of data in what I’ve discussed, for one thing (which makes it a weird thing for a “data scientist” to be trafficking in!). On the other hand, I do think that I have seen a few things that have value, and uncovered some unexamined assumptions that really need to be held up to sunlight. So the best I can think to do for now is blog them, and hopefully get the parts of the ideas that turn out not to be wrong into scientific discussions.
I’ve pretty much emptied the sack at this point, so I’m not planning to keep on writing more of these, unless I notice anything else. What I’d like to do today is draw up a coherent summary of where we stand with respect to breakthroughs, vaccines, Omicron, and the state of epidemic surveillance, drawing on the last four posts.
Why Do We Need A “False Breakthrough” Type, Anyway?
In the first post of this series, I introduced a typology of breakthrough infections that helped me organize the rest of this discussion. If you review that typology now (go ahead, I’ll wait) there is something about it that ought to strike you as a bit weird. Why on Earth should there even exist a Type 3a, “False Breakthrough Infection” category? How could it have come about that we are arguing about whether or not an infection is, in fact, an infection?
There is actually something unusual about the clinical status of a COVID-19 diagnosis: rather than being assessed on the basis of a constellation of symptoms, as is the case for most diseases, a case is declared a case on the base of a laboratory test, almost always a “Reverse Transcriptase Polymerase Chain Reaction” (RT-PCR) assay. And an RT-PCR test is a very sensitive test, capable of detecting very low levels of virus particle concentration (AKA “viral load”). It’s actually very impressive that a laboratory technology such as PCR, which would ordinarily require PhD-trained scientists to understand and operate, was rolled out so quickly and widely, and is now operated by hundreds of thousands of hourly-salary technicians, millions of times per day, with extremely low error rates. This is one of the scientific responses to the pandemic that went very right.
With every benefit there is a cost, however. In this case, the ability to detect extremely low levels of viral load collides with the characteristic response of a correctly-functioning, vaccine-primed, SARS-CoV-2-aware human immune system. As we have already discussed, when a human organism in possession of such an immune system is attacked by the virus, the virus does not explode on contact or bounce off some kind of impenetrable armor. Instead, the virus gains entry and begins to infect cells and reproduce itself. Early on in the process, however, the immune system becomes aware of the infection and duly moves to shut it down. What level of peak viral load can be attained before the response gains the upper hand is a matter of competing rates—rate of viral growth versus rate of various immune system infection-clearing processes. With the original SARS-CoV-2 strain, and some of the early prominent variants, it was very clearly the case that a lot of “breakthrough” infections that people were panicking over, and which were ostensibly lowering estimates of vaccine protective effectiveness against infection, were actually cases of perfectly normal vaccine-primed immune systems doing their jobs without a fuss, and of RT-PCR test ringing up “Positive” based on the very low viral loads characteristic of a failed infection. Hence, Type 3a, “False Breakthrough”.
But wait. Something’s not right. Why is this super-sensitive assay being fooled?
In my previous post, on Omicron’s actual status in the typology of breakthrough infections, I alluded near the end to a fact that strikes me as requiring much more of an explanation than is usually given. The fact in question is that in the SARS-CoV-2 epidemic, every time a new, rapidly-reproducing variant has burst on the scene, within a few months it has driven all its rival variants clean out of the community-spread genome.
Take a look at The Covariants.org Per-Country page, and let your mouse scroll over the United Kingdom chart—the UK has been consistently sequencing more specimens more assiduously, completely, and regularly and since far earlier than any other nation, as you can see from the number of sequences (the “num seq” pop-up figure), so it’s the best case study. You can see that until 12 October 2020 there was a variant winningly named “EU1” cruising to dominance over its competitors. But on 14 September something new had happened: 3 specimens had turned up with a new variant, named “Alpha”. By 8 March, Alpha has secured 98% of the circulating genome (34648/35670 specimens) and appeared on its way to crushing EU1 (173/35670 specimens), but again, something new had just happened: 6 specimens of a new variant, “Delta” had just shown up. You already know how this story goes: Delta swept the board. By Mid-August, Alpha sightings were as common as Elvis sightings (21/75887), and EU1 sightings were like unicorn sightings (2/75887). In the 1 November data—just prior to Omicron’s appearance—out of 96120 specimens only 9 were not Delta or some cousin of Delta. At that level, to explain the non-Delta signal, we’re really looking at accidents rather than spread: things that interfere with good mixing, such as small, isolated communities perhaps, or travel from distant areas. Natural alternatives to Delta had clearly been driven out of the larger circulating SARS-CoV-2 genome by the time Omicron showed up.
How Can We Know What Kind Of Breakthrough Infections Omicron Produces?
At the end of my previous post on breakthrough infections I suggested that there are in fact very good reasons to believe that the Omicron variant is not creating “dangerous” (to the patient) breakthrough infections, that is, it is not creating “Type 1 breakthrough infections”, in the typology that I set out in that post. The ability to create Type 1 breakthrough infections would make a variant very dangerous, because an infection by such a variant would evade vaccine-primed human immunity, and the patient’s immune system would have to start from scratch on the time-consuming process of learning to identify the virus, and to create the armament of antibodies and immune cells to fight it without the assistance of a vaccine. The “Type 3” breakthroughs (whether of the “false breakthrough” or “semi-breakthrough” sub-types) are much less dangerous. Neither sub-type actually evades vaccine-primed immune response: they merely appear to, because the rapid reproduction of the virus in the body leads to measurable, often infective levels of viral load despite neutralizing antibodies’ efforts to restrain that growth; but the infection’s early doom is already sealed, because the cellular part of the adaptive immune system—specifically the killer T-cells—are on their way, and will wipe it out in short order. So while the infected individual may be infective (Type 3b), he or she is not usually at risk of severe disease.
This is an orderly proceeding for an immune response to an infection, incidentally. Neutralizing antibodies are only the first layer of the adaptive immune system, and despite their prominence in media discussion of vaccines and therapies they are not responsible for either preventing severe disease or clearing an infection. Antibodies merely slow down the rate of growth of the infection, buying time for the real heavy hitters—the T-Cells—to be mobilized to fight off the infection. That’s the key difference between Type 1 and Type 3: if a variant can create Type 1 breakthroughs, the T-Cells can’t fight it, whereas if it can only create Type 3 breakthroughs, the T-Cells will kick the crap out of it.
So it actually matters what kind of breakthrough infections Omicron is producing, and that’s the reason I’m trying to create some badly-needed clarity in the discussion surrounding the wretchedly ill-chosen term “breakthrough”. What, then, is the evidence that Omicron is not a Vaccine Escape Variant of Concern?
It’s amazing to me how much damage a badly-chosen scientific term can do, in a high-consequence scientific field such as COVID-19 epidemiology. The term “Breakthrough infection”, which quickly filtered from journal literature on vaccine effectiveness to public media, turns out to be so poorly defined that it even confuses scientific discussions, and when it enters public discourse it engenders mostly misinformation and panic. It is downright daft terminology, which is unfortunately as ineradicable now as the virus itself. In this post, I’d like to at least try to fix it a little, so the damn term can do some useful work for a change.
I haven’t posted a lot about vaccine efficacy lately, largely because the frenzy of vaccine development and clinical trial results basically slowed way down around June 2021, and there hasn’t been that much to write about since on the subject. I’ve been thinking about what to do with this site ever since. Some people do seem to find it useful—there have been nearly 25,000 visitors from all over the world since I started writing about vaccine efficacy, and I hope that those people found information that was valuable to them. To the extent that they did, it makes me somewhat proud, since I am a statistics person rather than a clinically-trained person, so having an impact in the COVID-19 pandemic, however small, feels like an achievement. On the other hand, that same lack of clinical training means that I have to watch myself so as to write things that are justified by data, and keep from making wild, poorly-informed statements that do more harm than good. I feel that so far I’ve stayed on the right side of that line.
I may risk that balance in the next phase of this blog’s development.
I plan to write a few observations of my own on the state of epidemic surveillance, epidemic modeling, and epidemic data, specifically with respect to the COVID-19 epidemic. I am doing this in part because I’ve been more deeply involved in data-driven epidemiological work, especially with respect to vaccine effectiveness (different from efficacy, because it characterized real risk reduction in real populations, rather than “pure” clinical properties of vaccines), culminating in a paper demonstrating the possibilities of large-scale data analytics for epidemic surveillance and vaccine assessment. In the process, I have become somewhat frustrated with the state of data curation and availability, but also with some of the model-premises underlying discussions of subjects such as vaccines, “breakthrough” infections, variants and their potential for vaccine escape, and so on. In my opinion there is a great deal of intellectual confusion about these terms and concepts, and this confusion is feeding needless media and policy panic (and occasionally distracting from necessary panic). I feel I need a place to write down everything that I feel is (usually) subtly or (occasionally) grossly wrong about the public and scientific discussions of these issues. And I happen to have a more-or-less epidemiological blog. So I might as well do it here.
The cost of this change of direction is that I doubt that I can maintain the careful stance of defensible scientific statements that I tried to keep this blog to so far. Quasi-editorials on epidemiology by a statistically well-informed but barely-clinically-literate observer of the field should by no means be taken as authoritative refutations of anything, or in fact as anything more than spurs for further discussion by people working in the field, with whom I would be delighted to engage, and be told in exhausting detail all the reasons why what I’m writing is wrong-headed. I do listen, and try to learn. But I will also argue. I feel that I will have accomplished something useful if I at least bring to light a few unexamined or under-examined assumptions, and occasion a fruitful discussion of those assumptions, even if in the end I am the only one who feels educated by the process.
Nonetheless, I have a strong suspicion that I’ve seen some real issues—defects in how clinical data is created and curated and made available, defects of modeling, catastrophic terminological confusion—that need to be brought into the light. I’ll be discussing these in a series of posts.
Anotherpress release by Pfizer, this one with results from their main phase 3 trial (46,307 participants). They have followed up participants for 6 months post shot-2. The press release has some slices of data, which will certainly be released in more detail at the time of their application for full FDA approval — and presumably when they submit a journal article.
Here are the two things that caught my eye:
Pfizer/BioNTech, phase 3 study, shot 2 + 6 months follow-up. Left panel: overall efficacy. Right panel: South Africa results for confirmed infections by B.1.351 variant. For more information about the features of these plots, seethis post.
On the left we see more confirmation of what we knew: the superb efficacy of the Pfizer/BioNTech COVID-19 vaccine. There is now so much data that the uncertainty on the efficacy is tiny. The peak posterior probability estimate in the plot is 91%. The result is basically consistent with the previous results. It’s worth remembering that the original targets for vaccine efficacies when the development effort began in early 2020 was “anything over 50%.” The fact that this generation of vaccines is ringing the bell in the 70-90% range is really an astonishing feat.
Now to the right-hand panel. The trial has a South Africa arm (well, at 800 participants, more like a “finger”). Of these, 9 placebo-group participants became infected, and zero vaccine group-participants. Of the 9 placebo-group participants, 6 were confirmed to have been infected by the B.1.351 lineage. If you have 6 placebo infections, and zero vaccine infections, you get the plot on the right hand side. This is what the flacks who work for Pfizer’s PR department have to say about this:
In South Africa, where the B.1.351 lineage is prevalent and 800 participants were enrolled, nine cases of COVID-19 were observed, all in the placebo group, indicating vaccine efficacy of 100% (95% CI, [53.5, 100.0]). In an exploratory analysis, the nine strains were sequenced and six of the nine were confirmed to be of the B.1.351 lineage. These data support previous results from immunogenicity studies demonstrating that BNT162b2 induced a robust neutralizing antibody response to the B1.351 variant, and although lower than to the wild-type strain, it does not appear to affect the high observed efficacy against this variant.i
OK, now look at that plot. Does that say “100% efficacy against B.1.351” to you? Because to me it says “we got bupkus for evidence, since we had too few people in South Africa participating in this study.” The 90% credible region starts at 50% efficacy!
You should also be aware that the stuff about “robust neutralizing antibody response” is based on this paper, which however showed that in-vitro neutralization of B.1.351 required far more concentrated vaccine-induced antibodies than were required for other variants — I cannot understand how the how one could possibly characterize that result as “robust”. In any event, its a wet lab result, not a clinical trial. We still don’t have any quality evidence about the performance of the mRNA vaccines against the B.1.351 variant. And that has me very worried.
There’s a new journal article at the New England Journal of Medicine, detailing a Phase 1b/2 clinical trial of the AstraZeneca COVID-19 vaccine, conducted in South Africa. The principal thrust of the article is to gauge efficacy against the B.1.351 virus variant now prevalent in South Africa. The news is…not good:
Efficacy of the AstraZeneca COVID-19 vaccine against the B.1.351 variant. For an explanation of the features in this plot, seethis post.
This plot shows the efficacy of the vaccine, restricted to the sample of patients who developed “mild or moderate” COVID-19 (there were no “severe” cases in this small dataset) and were confirmed infected by the B.1.351 variant of SARS-CoV-2. Basically, there is no evidence of protective efficacy against the B.1.351 variant.
This is not to pile on to AstraZeneca, which is having a bad week anyway due to safety concerns. So far, no current COVID-19 vaccine has demonstrated acceptable efficacy against this particular variant (as opposed to the B.1.1.7 “UK” variant, or the P.1 “Brazil” variant). The Johnson & Johnson and Novavax vaccines also had disappointing performance against this variant. Collecting the three South Africa studies together for comparison, we see this:
Currently available South Africa-based clinical trials. Left panel: J&J. Middle panel: Novavax. Right panel: AstraZeneca.
None of these looks particularly good. Part of the reason the AZ efficacy plot looks much worse than those corresponding to the J&J and Novavax South Africa studies is that those studies had about 6 times more data, so the evidence for what protection their vaccines can offer is better than AZ’s evidence — the protection itself might be about the same. Also note that the J&J and Novavax studies did not single out B.1.351 cases, so there are about 5% “classic” SARS-CoV-2 cases in the above plots, and this necessarily improves their apparent efficacy with respect to the AstraZeneca result. Another confounding issue is that because of the incompetence of AstraZeneca’s vaccine manufacturing (which also led to the data issues that I discussed here), the administered doses were not all the same — the article states that vaccine recipients in the trial received “…a 0.33-to-0.5-ml dose (depending on the lot) of the ChAdOx1 nCoV-19 vaccine”, which means they don’t even have a well-controlled idea of how much vaccine they administered in this study.
Note also there is in vitro evidence that the Pfizer and Moderna vaccines will offer at best weak protection against B.1.351 (see here for Pfizer and here for Moderna).
So the story of vaccine-mediated resistance to this variant remains the same as I discussed in this post. At the moment, there is weak evidence for protection against severe disease caused by B.1.351, and no evidence at all for protection against transmission. We’re still on-track for an urgent program of B.1.351-specific vaccine booster shots (Moderna has one already in trials) as soon as they can be developed, approved, and produced. In my opinion the need for such boosters will be clear by late Summer 2021, and getting urgent by Fall.
Novavax issued another press release today, with some partial information concerning the now-completed trials, a Phase 3 trial in the UK, and Phase 2b trial in South Africa. The data is a continuation of the data analyzed in this post, and tells a similar story, although with more abundant numbers (and hence less uncertainty). There is some actual data in the press release, but there are also some efficacy estimates not backed by released data, which I can therefore not evaluate here. As usual, when the Emergency Use Authorization request gets to the FDA, a briefing document will be made public with complete trial information. I will issue another update when that happens.
Meantime, here’s what the released data for UK trial looks like:
Results from the Novavax Phase 3 UK trial. Left panel: Overall efficacy. Right panel: Efficacy against “severe” disease. For an explanation of the features in these plots, see thispost.
This looks pretty good. Not that different from the preliminary results, but the constraints on efficacy are tighter, since the numbers are higher. The overall result in the left panel is extremely good (most likely value 90%), despite high prevalence of the B.1.1.7 variant. That’s the variant that is more transmissible, and possibly responsible for more severe disease with worse clinical outcomes, but which is also known not to be a so-called “escape variant” — that is, it is not less susceptible to immunity due to previous infection by “classic” SARS-CoV-2, or due to existing vaccines. So it’s not surprising that the Novavax vaccine stops it.
The “severe” disease result is shown on the right. This is what, unfortunately, is described in the press release and reported in naive media outlets as “100%” protection. From the plot it should be clear why this is an inappropriate description of the result — when you have 5 severe cases in the placebo group and 0 in the vaccine group, that’s good news, but nowhere near establishing “100% protection” — based on this data, the 90% credible region says that the protection could be as low as 40%! This kind of thing is the why it is useful to reason in terms of quantified uncertainties, as displayed in these plots, rather than with naive and deceptive point estimates. What we can say based on this data is that there is some moderately weak evidence of efficacy greater than about 40% against severe disease. If Novavax had designed their study as well as Moderna did, including many individuals at risk for severe disease, then they might have collected better evidence (as Moderna did).
Now, here’s the South African trial data analysis:
Results from the Novavax Phase 2b South Africa trial. Left panel: Overall efficacy. Right panel: Efficacy against “severe” disease.
In the overall results on the left we see again the troubling effect of the South African “escape variant”, B.1.351, which has already demonstrated its ability to evade immunity produced by the Johnson & Johnson and AstraZeneca vaccines, and for which there is in vitro evidence of decreased immunogenesis by the Pfizer/BioNTech and Moderna vaccines. This result is no worse than the J&J South Africa result, but it’s not any better either. Despite the spin in the press release, it suggests that efficacy to the B.1.351 variant is likely lower than the 50% threshold limit established by the FDA for usefulness. And if the vaccines are this poor at stopping disease, they are almost certainly nearly ineffective at stopping the spread of the virus, since it is being propagated by all all the carriers who get mild or moderate forms of the disease, to say nothing of all the asymptomatic carriers.
This reinforces the point that I made in this post: The B.1.351 variant very likely largely escapes immunity from all existing vaccines. Since (1) we can expect most of the Northern hemisphere to be vaccinated by early summer, so that other variants will be under control, and (2) we know that it only takes a few months for SARS-CoV-2 to travel all over the globe (witness this time it took for the epidemic to spread from Wuhan to the rest of the world in the first place), we can expect that most COVID-19 in the Northern hemisphere will be caused by B.1.351 by sometime this year. Booster vaccines that address B.1.351 are already being prepared — Moderna’s is already in Phase 2, and there’s no doubt that the other manufacturers aren’t far behind. So, we should all get shots as soon as they are available, because they are effective at stopping the spread of classic SARS-CoV-2 and most of its variants. But we should all be prepared to get booster shots this year, as soon as the B.1.351-specific vaccines are available.
The FDA’s approval of Emergency Use Authorization of the Johnson & Johnson COVID-19 vaccine is being greeted ecstatically and uncritically by public officials and media. In my opinion, this is an indication that hardly anyone has actually looked at the data supplied by the company to the EUA committee, or read the briefing document in detail.
I wrote up a quick but careful analysis last week, which you may review here.
For present purposes, I’ll note only in passing the statistical illiteracy of journalistic celebrations of “85 percent efficacy against severe forms of COVID-19 and 100 percent efficacy against hospitalization and death” (NYT), figures that ignore both the considerable uncertainties in efficacy estimates and the carefully cherry-picked data that produces such numbers by nimble selections of geographical subregions of the trial, and by passing back and forth between results for 14 days and 28 days post-vaccine, whichever produces the better-seeming (though not actually better) result.
The point that I wish would draw notice is that the South African arm of the trial suggests that a variant crisis is imminent. The J&J vaccine is marginally effective at stopping transmission of the B.1.351 variant that is 95% prevalent in South Africa, and it’s efficacy against the severe disease could actually be as low as 50%. Here are the two relevant plots from that arm of the trial:
South Africa results. Left panel: Protection against “moderate or severe” disease. Right panel: Protection against “severe” disease.For an explanation of the features of these plots, seethis post.
The right-hand figure offers some reassurance about “severe” disease (although if you think it means 80% you should not work as a science reporter). The left-hand figure, however, is extremely worrying. It says that the vaccine does nothing to restrain the spread of the virus, since all those “moderate” COVID-19 cases are shedding virus, and so are a nearly equal number of asymptomatic cases.
Here are some consequences:
B.1.135 Is The West’s Future Native Variant
It took less than 3 months for the original SARS-CoV-2 virus to travel from Wuhan to the U.S. and become an uncontrolled epidemic. It’s probably safe to assume that the freely-propagating B.1.135 variant, unrestrained by vaccination, will soon have high prevalence in the US and in Europe as well. It has mutated its way to a fitness advantage, so it will likely take over the SARS-CoV-2 genome within in a few months, probably by June at the latest.
It’s Not Just J&J’s Problem
I have no beef against J&J, and don’t mean to single the company out for criticism. There is plenty of evidence now that this is not just a J&J vaccine issue. We already know that the Novavax clinical trial had a South Africa arm that found the same thing. And South Africa has halted it’s rollout of the AstraZeneca vaccine, asserting that it “…offers minimal protection against mild and moderate cases”, as a consequence of the variant’s prevalence.
We don’t know how B.1.135 affects the efficacy of the Pfizer and Moderna vaccines, or that of the Sputnik V vaccine for that matter, since their clinical trials saw essentially zero prevalence of that variant. One may hope that since those vaccines appear to induce a more robust immune response, they may yield broader immunity against variants. But without new trials, all we can do is wait and find out.
Transmission Matters
Much of the media celebration appears to center on the ability of the J&J vaccine to prevent “severe” disease. As we saw above, even this ability is reduced in light of the advent of B.1.135. But ignoring the ineffectiveness of the vaccine at preventing transmission is madness. The probability of a mutation that increases transmissibility (or disease severity, for that matter) is proportional to the infection rate — the more cases per day, the higher the chances of a more dangerous mutation. The reason that we’re seeing proliferation of variants now, to an extent not noticeable during the March or August waves, is that the Winter wave case count eclipsed that of the other two — there have been many more mutation opportunities since October 2020 than there were before.
A vaccine is a tool for epidemic control, not a treatment or a cure. Focusing on prevention of the worst disease outcomes while ignoring transmission prevention totally misses the point. It’s tantamount to celebrating relief now, while ignoring the worse trouble to come.
A Silver Lining
The one reason that I can see not to panic now is that the next-generation vaccines that have come into being in the past year — mRNA and adenovirus vector — are reportedly very straightforward to re-target at new variants, and updated versions can be quickly produced at industrial scales, at least after current supply-chain bottlenecks are sorted out. What this means is that while everyone who gets a J&J, AZ, or Novavax vaccine now will be needing a booster by summer (and this may also be true for the Pfizer, Moderna, and Sputnik vaccines as well), at least there is reason to believe that such boosters will be widely available. For this reason, the public-health messaging on the J&J vaccine is correct: if you are offered a shot of the J&J vaccine, you should accept it. Just make sure you get a booster as soon as it is available.
The epidemic control strategy has to be to get total infection rates down (to slow the rise of newer, more dangerous variants), while rapidly developing, producing, distributing, and jabbing updated boosters to stamp out existing variants. And doing this everywhere, including in third-world nations that can’t afford to fund their own rapid-response anti-variant campaigns, because any large reservoir of transmitted virus is a potential source of newer, even more dangerous variants.
Moreover, the summer-booster messaging needs to start now. We all know how delicate public support for mass vaccination campaigns is, and how easily it can be poisoned by militantly ignorant, scientifically illiterate disinformation spread by the anti-vaxxer crowd. Those people will have a field day with rising-again infection rates among vaccinated individuals that are to be expected this summer and fall, unless there’s a comprehensive public health messaging program about variants and boosters. The sooner the better.
And reporters need to get smarter about reporting “victories” like this one, that carry seeds of future problems. When the victory turns out to be not as complete as first reported, and possibly even reversible, the consequences for public support of mass vaccination campaigns are unlikely to be good.
