Peter Doshi’s coronavirus vaccine safety pieces

Peter Doshi has recently published pertinent coronavirus vaccine safety pieces in the British Medical Journal (BMJ).

He is an associate editor at The BMJ and on the News & Views team. Based in Baltimore, he is also an assistant professor of pharmaceutical health services research at the University of Maryland School of Pharmacy.

His research focuses on policies related to drug safety and effectiveness evaluation in the context of regulation, evidence-based medicine, and debates over access to data.

His two BMJ latest pieces are:

1. Pfizer and Moderna’s “95% effective” vaccines—let’s be cautious and first see the full data, BMJ Nov 26, 2020
2. Will covid-19 vaccines save lives? Current trials aren’t designed to tell us, BMJ Oct 21, 2020

These have been reformatted below to be faster and easier to consume.

If you don’t know what viral vectors are, check out this podcast I did featuring Liz Parrish.

For those not familiar with relative vs absolute risk, let me quickly explain. Absolute refers to your own personal risk (absolute difference) and is the key metric, not relative. If the 5-year risk for heart attack is 2 in 100 (2%) in a group of patients treated conventionally and 1 in 100 (1%) in patients treated with the new drug, the absolute difference is derived by simply subtracting the two risks:

2% – 1% = 1%.

The relative difference is the ratio of the two risks. Given the data above, the relative difference is:

1% ÷ 2% = 50%

Expressed as a relative difference, the new drug reduces the risk for heart attack by half.

Pfizer and Moderna’s “95% effective” vaccines—let’s be cautious and first see the full data

Published BMJ, November 26, 2020

In the United States, all eyes are on Pfizer and Moderna. The topline efficacy results from their experimental covid-19 vaccine trials are astounding at first glance. Pfizer says it recorded 170 covid-19 cases (in 44,000 volunteers), with a remarkable split: 162 in the placebo group versus 8 in the vaccine group. Meanwhile Moderna says 95 of 30,000 volunteers in its ongoing trial got covid-19: 90 on placebo versus 5 receiving the vaccine, leading both companies to claim around 95% efficacy.

Let’s put this in perspective.

1. A relative risk reduction is being reported, not absolute risk reduction, which appears to be less than 1%.
2. These results refer to the trials’ primary endpoint of covid-19 of essentially any severity, and importantly not the vaccine’s ability to save lives, nor the ability to prevent infection, nor the efficacy in important subgroups (e.g. frail elderly). Those still remain unknown.
3. These results reflect a time point relatively soon after vaccination, and we know nothing about vaccine performance at 3, 6, or 12 months, so cannot compare these efficacy numbers against other vaccines like influenza vaccines (which are judged over a season).
4. Children, adolescents, and immunocompromised individuals were largely excluded from the trials, so we still lack any data on these important populations.

I previously argued that the trials are studying the wrong endpoint, and for an urgent need to correct course and study more important endpoints like prevention of severe disease and transmission in high risk people.

Yet, despite the existence of regulatory mechanisms for ensuring vaccine access while keeping the authorization bar high (which would allow placebo-controlled trials to continue long enough to answer the important question), it’s hard to avoid the impression that sponsors are claiming victory and wrapping up their trials (Pfizer has already sent trial participants a letter discussing “crossing over” from placebo to vaccine), and the FDA will now be under enormous pressure to rapidly authorize the vaccines.

But as conversation shifts to vaccine distribution, let’s not lose sight of the evidence. Independent scrutiny of the underlying trial data will increase trust and credibility of the results. There also might be important limitations to the trial findings we need to be aware of.

Most crucially, we need data-driven assurances that the studies were not inadvertently unblinded, by which I mean investigators or volunteers could make reasonable guesses as to which group they were in.

Blinding is most important when measuring subjective endpoints like symptomatic covid-19, and differences in post-injection side-effects between vaccine and placebo might have allowed for educated guessing.

Past placebo-controlled trials of influenza vaccine were not able to fully maintain blinding of vaccine status, and the recent “half dose” mishap in the Oxford covid-19 vaccine trial was apparently only noticed because of milder-than-expected side-effects. (And that is just one of many concerns with the Oxford trial.)

In contrast to a normal saline placebo, early phase trials suggested that systemic and local adverse events are common in those receiving vaccine. In one Pfizer trial, for example, more than half of the vaccinated participants experienced headache, muscle pain and chills—but the early phase trials were small, with large margins of error around the data. Few details from the large phase 3 studies have been released thus far.

Moderna’s press release states that 9% experienced grade 3 myalgia and 10% grade 3 fatigue; Pfizer’s statement reported 3.8% experienced grade 3 fatigue and 2% grade 3 headache. Grade 3 adverse events are considered severe, defined as preventing daily activity. Mild and moderate severity reactions are bound to be far more common.

One way the trial’s raw data could facilitate an informed judgment as to whether any potential unblinding might have affected the results is by analyzing how often people with symptoms of covid-19 were referred for confirmatory SARS-CoV-2 testing. Without a referral for testing, a suspected covid-19 case could not become a confirmed covid-19 case, and thus is a crucial step in order to be counted as a primary event: lab-confirmed, symptomatic covid-19. Because some of the adverse reactions to the vaccine are themselves also symptoms of covid-19 (e.g. fever, muscle pain), one might expect a far larger proportion of people receiving vaccine to have been swabbed and tested for SARS-CoV-2 than those receiving placebo.

This assumes all people with symptoms would be tested, as one might expect would be the case. However the trial protocols for Moderna and Pfizer’s studies contain explicit language instructing investigators to use their clinical judgment to decide whether to refer people for testing. Moderna puts it this way:

“It is important to note that some of the symptoms of COVID-19 overlap with solicited systemic ARs that are expected after vaccination with mRNA-1273 (eg, myalgia, headache, fever, and chills). During the first 7 days after vaccination, when these solicited ARs are common, Investigators should use their clinical judgement to decide if an NP swab should be collected.”

This amounts to asking investigators to make guesses as to which intervention group patients were in. But when the disease and the vaccine side-effects overlap, how is a clinician to judge the cause without a test? And why were they asked, anyway?

Importantly, the instructions only refer to the first seven days following vaccination, leaving unclear what role clinician judgment could play in the key days afterward, when cases of covid-19 could begin counting towards the primary endpoint. (For Pfizer, 7 days after the 2nd dose. For Moderna, 14 days.)

In a proper trial, all cases of covid-19 should have been recorded, no matter which arm of the trial the case occurred in. (In epidemiology terms, there should be no ascertainment bias, or differential measurement error). It’s even become common sense in the Covid era: “test, test, test.” But if referrals for testing were not provided to all individuals with symptoms of covid-19—for example because an assumption was made that the symptoms were due to side-effects of the vaccine—cases could go uncounted.

Data on pain and fever reducing medicines also deserve scrutiny. Symptoms resulting from a SARS-CoV-2 infection (e.g. fever or body aches) can be suppressed by pain and fever reducing medicines. If people in the vaccine arm took such medicines prophylactically, more often, or for a longer duration of time than those in the placebo arm, this could have led to greater suppression of covid-19 symptoms following SARS-CoV-2 infection in the vaccine arm, translating into a reduced likelihood of being suspected for covid-19, reduced likelihood of testing, and therefore reduced likelihood of meeting the primary endpoint. But in such a scenario, the effect was driven by the medicines, not the vaccine.

Neither Moderna nor Pfizer have released any samples of written materials provided to patients, so it is unclear what, if any, instructions patients were given regarding the use of medicines to treat side effects following vaccination, but the informed consent form for Johnson and Johnson’s vaccine trial provides such a recommendation:

“Following administration of Ad26.COV2.S, fever, muscle aches and headache appear to be more common in younger adults and can be severe. For this reason, we recommend you take a fever reducer or pain reliever if symptoms appear after receiving the vaccination, or upon your study doctor’s recommendation.”

There may be much more complexity to the “95% effective” announcement than meets the eye—or perhaps not. Only full transparency and rigorous scrutiny of the data will allow for informed decision making. The data must be made public.

Peter Doshi, associate editor, The BMJ.

Will covid-19 vaccines save lives? Current trials aren’t designed to tell us

The world has bet the farm on vaccines as the solution to the pandemic, but the trials are not focused on answering the questions many might assume they are. Peter Doshi reports

Published BMJ, October 21, 2020

As phase III trials of covid-19 vaccines reach their target enrolments, officials have been trying to project calm. The US coronavirus czar Anthony Fauci and the Food and Drug Administration leadership have offered public assurances that established procedures will be followed. Only a “safe and effective” vaccine will be approved, they say, and nine vaccine manufacturers issued a rare joint statement pledging not to prematurely seek regulatory review.

But what will it mean exactly when a vaccine is declared “effective”? To the public this seems fairly obvious. “The primary goal of a covid-19 vaccine is to keep people from getting very sick and dying,” a National Public Radio broadcast said bluntly.

Peter Hotez, dean of the National School of Tropical Medicine at Baylor College of Medicine in Houston, said,

“Ideally, you want an antiviral vaccine to do two things . . . first, reduce the likelihood you will get severely ill and go to the hospital, and two, prevent infection and therefore interrupt disease transmission.”

Yet the current phase III trials are not actually set up to prove either (table 1). None of the trials currently under way are designed to detect a reduction in any serious outcome such as hospital admissions, use of intensive care, or deaths.
Nor are the vaccines being studied to determine whether they can interrupt transmission of the virus.

October 21, 2020

“None of the trials currently under way are designed to detect a reduction in any serious outcome such as hospital admissions, use of intensive care, or deaths.

Nor are the vaccines being studied to determine whether they can interrupt transmission of the virus.”

Peter Doshi

Editor at The BMJ

Evaluating mild, not severe, disease

In a September interview Medscape editor in chief Eric Topol pondered what counts as a recorded “event” in the vaccine trials. “We’re not talking about just a PCR [polymerase chain reaction test]-positive mild infection. It has to be moderate to severe illness to qualify as an event, correct?” he asked.

“That’s right,” concurred his guest, Paul Offit, a vaccinologist who sits on the FDA advisory committee that may ultimately recommend the vaccines for licence or emergency use authorisation.

But that’s not right. In all the ongoing phase III trials for which details have been released, laboratory confirmed infections even with only mild symptoms qualify as meeting the primary endpoint definition. In Pfizer and Moderna’s trials, for example, people with only a cough and positive laboratory test would bring those trials one event closer to their completion. (If AstraZeneca’s ongoing UK trial is designed similarly to its “paused” US trial for which the company has released details, a cough and fever with positive PCR test would suffice.)

Part of the reason may be numbers. Severe illness requiring hospital admission, which happens in only a small fraction of symptomatic covid-19 cases, would be unlikely to occur in significant numbers in trials. Data published by the US Centers for Disease Control and Prevention in late April reported a symptomatic case hospitalisation ratio of 3.4% overall, varying from 1.7% in 0-49 year olds and 4.5% in 50-64 year olds to 7.4% in those 65 and over.

Because most people with symptomatic covid-19 experience only mild symptoms, even trials involving 30 000 or more patients would turn up relatively few cases of severe disease.

In the trials, final efficacy analyses are planned after just 150 to 160 “events,”—that is, a positive indication of symptomatic covid-19, regardless of severity of the illness.

Yet until vaccine manufacturers began to release their study protocols in mid-September, trial registries and other publicly released information did little to dispel the notion that it was severe covid-19 that the trials were assessing. Moderna, for example, called hospital admissions a “key secondary endpoint” in statements to the media.

And a press release from the US National Institutes of Health reinforced this impression, stating that Moderna’s trial “aims to study whether the vaccine can prevent severe covid-19” and “seeks to answer if the vaccine can prevent death caused by covid-19.”

But Tal Zaks, chief medical officer at Moderna, told The BMJ that the company’s trial lacks adequate statistical power to assess those outcomes. “The trial is precluded from judging [hospital admissions], based on what is a reasonable size and duration to serve the public good here,” he said.

Hospital admissions and deaths from covid-19 are simply too uncommon in the population being studied for an effective vaccine to demonstrate statistically significant differences in a trial of 30,000 people. The same is true of its ability to save lives or prevent transmission: the trials are not designed to find out.

October 21, 2020

“Hospital admissions and deaths from covid-19 are simply too uncommon in the population being studied for an effective vaccine to demonstrate statistically significant differences in a trial of 30,000 people.

Peter Doshi

Editor at The BMJ

Zaks said, “Would I like to know that this prevents mortality? Sure, because I believe it does. I just don’t think it’s feasible within the timeframe [of the trial]—too many would die waiting for the results before we ever knew that.”

Stopping transmission

What about Hotez’s second criterion, interrupting virus transmission, which some experts have argued should be the most important test in phase III studies?

“Our trial will not demonstrate prevention of transmission,” Zaks said, “because in order to do that you have to swab people twice a week for very long periods, and that becomes operationally untenable.”

He repeatedly emphasised these “operational realities” of running a vaccine trial. “Every trial design, especially phase III, is always a balancing act between different needs,” he said. “If you wanted to have an answer on an endpoint that happens at a frequency of one 10th or one fifth the frequency of the primary endpoint, you would need a trial that is either 5 or 10 times larger or you’d need a trial that is 5 or 10 times longer to collect those events. Neither of these, I think, are acceptable in the current public need for knowing expeditiously that a vaccine works.”

Zaks added, “A 30,000 [participant] trial is already a fairly large trial. If you’re asking for a 300 000 trial then you need to talk to the people who are paying for it, because now you’re talking about not a $500m to $1bn trial, you’re talking about something 10 times the size.

And I think the public purse and operational capabilities and capacities we have are rightly spent not betting the farm on one vaccine but, as Operation Warp Speed [the US government’s covid-19 vaccine plan] is trying to do, making sure that we’re funding several vaccines in parallel.”

Debating endpoints

Still, it’s fair to say that most of the general public assumes that the whole point of the current trials, besides testing safety, is to see whether the vaccine can prevent bad outcomes. “How do you reconcile that?” The BMJ asked Zaks.

“Very simply,” he replied. “Number one, we have a bad outcome as our endpoint. It’s covid-19 disease.” Moderna, like Pfizer and Janssen, has designed its study to detect a relative risk reduction of at least 30% in participants developing laboratory confirmed covid-19, consistent with FDA and international guidance.

Number two, Zaks pointed to influenza vaccines, saying they protect against severe disease better than mild disease. To Moderna, it’s the same for covid-19: if its vaccine is shown to reduce symptomatic covid-19, it will be confident it also protects against serious outcomes.

But the truth is that the science remains far from clear cut, even for influenza vaccines that have been used for decades. Although randomised trials have shown an effect in reducing the risk of symptomatic influenza, such trials have never been conducted in elderly people living in the community to see whether they save lives.

Only two placebo controlled trials in this population have ever been conducted, and neither was designed to detect any difference in hospital admissions or deaths. Moreover, dramatic increases in use of influenza vaccines has not been associated with a decline in mortality.

Telegram

Join My Telegram Channel

t.me/dryburgh

More Resources:

• James Lyons-Weiler — Pathogenic Priming: Coronavirus Vaccine Safety Warning
• Dr Yeadon’s (former Pfizer VP) Coronavirus Vaccine Safety Petition
• Dr. Larry Palevsky — Covid-19 Vaccine Safety Concerns