How Effective Are Flu Shots?

Once again, with the arrival of autumn, it's time for a flu shot.

I get a flu shot every year even though I know they’re not perfect. I figure they’re a relatively easy and inexpensive way to reduce the chance of having a miserable week.

I’ve heard on various news media that their effectiveness is about 60%. But what does 60% effectiveness mean, exactly? How much does this actually reduce the chances that I’ll get the flu in any given year? I'm going to explore this and go beyond the news media simplification and present you with very clear answers to these questions. Quite frankly, some of the results were not what I expected.

We’ll Find Our Answers in Randomized, Controlled Trials (RCTs)

I’m a numbers guy. I use numbers to understand the world. My background is in research, so when I want to understand an issue, I look at the primary research. If I can understand the researchers’ methodology, the data they collect, and how they draw their conclusions, I’ll understand the issue at a deeper, more fundamental level than news reports typically provide.  

To understand flu shot effectiveness, I’m only going to assess double-blind, randomized controlled trials, the gold standard. These studies are more expensive to conduct but provide better results than observational studies. (I discuss the differences between these two types of studies in my post about the benefits of vitamins.)

The two influenza vaccination studies I’ll look at satisfy the above criteria and are listed in a section of references for health professionals on the CDC’s website. Presumably these studies make a good case, using trusted data. Along the way, we’ll use Minitab statistical software to analyze their data for ourselves.

Defining the Effectiveness of Flu Shots

Flu shots contain vaccine for three influenza viruses that researchers predict will be the most common in a given flu season. However, plenty of other viruses (flu and otherwise) also are circulating and can make you sick. Many illnesses with flu-like symptoms are incorrectly attributed to the flu.

Consequently, the best studies use a lab to identify the specific virus that infects each of their sick subjects. These studies only count the subjects with confirmed cases of the three types of influenza virus. Effectiveness is defined as the reduction in these three influenza viruses among those who were vaccinated compared to those who were not vaccinated.

The Two Studies of the Flu Vaccine

It’s time to dig into the data! For me, this is where it gets exciting. You can hear about effectiveness on TV, but this is where it all comes from: counts of sick people in the experimental groups.

The Beran Study

The Beran et al. study1 assesses the 2006/2007 flu season and tracks its subjects from September to May. Subjects in this study range from 18-64 years old.

Treatment

Flu count

Group size

Shot

49

5103

Placebo

74

2549

Because we want to compare the proportions between two groups, we’ll use the Two Proportions test in Minitab. To do this yourself, in Minitab, go to Stat > Basic Statistics > 2 Proportions. In the dialog, choose Summarized data and enter the data from the table above. Click OK, and you get the results below:

The p-value of 0.000 tells us that there is a significant difference between the two groups. The estimated difference between the vaccinated group and the placebo group is 1.9 percentage points. Because this is an RCT, it's fairly safe to assume that the vaccination caused the difference between the groups. However, outside of a randomized experiment, it's not wise to assume causality.

The vaccine effectiveness (or efficacy) is a relative reduction in risk between the two groups. You simply take the relative risk ratio of (vaccinated proportion/unvaccinated proportion) and subtract that from 1. We can get the proportion for each group from the Sample p column in Minitab’s output:

1 - (0.009602/0.029031) =  0.669

This study finds a 66.9% vaccine efficacy for the flu shot compared to the placebo.

The Monto Study

The Monto et al study2 assesses the 2007-2008 flu season and tracks its subjects from January to April. Subjects in this study range from 18-49 years old.

Treatment

Flu count

Group size

Shot

28

813

Placebo

35

325

We’ll do the Two Proportions test again for this study. This time, enter the numbers from the above table into the dialog.

Again, the p-value indicates that there is a significant difference between the two groups. The estimated difference between the vaccinated group and the placebo group is 7.3 percentage points. Let's calculate the effectiveness:

1 – (0.034440/0.107692) = 0.680

This study finds a 68.0% vaccine efficacy for the flu shot compared to the placebo.

Conclusions So Far

We’ve looked at the data from two gold-standard studies and have drawn the same conclusions that you commonly hear on the news. Flu shots significantly reduce the number of influenza infections, and they are about 68% effective.

However, looking at the data and analyses myself, I have new insights. Specifically, the low number of influenza cases in the placebo group for each study caught my eye, and that’s what we’re looking at next.

What It Means for You: Relative versus Absolute Risk

If you’re like me, the 68% effective statistic isn’t too helpful. The problem is that it is a relative comparison of risk, not an absolute assessment of risk. To illustrate the difference, consider which type of assessment is more useful:

1. Relative assessment: Your car is travelling half as fast as another car, but you don’t know the true speed of either car.
    
2. Absolute assessment: Your car is travelling at 30 MPH and the other car is travelling at 60 MPH.

Clearly, #2 is much more useful. Similarly, it would be more helpful to know the absolute risk of catching the flu if you get the shot versus not getting it!

Vaccine effectiveness is a relative risk

Vaccine effectiveness doesn’t tell you the exact risk of catching the flu for either group. Instead, it involves dividing one proportion by the other for the relative risk. In fact, as you should recall, effectiveness is the inverse of the relative risk, which makes it even harder to interpret. 67% effectiveness indicates that a vaccinated person has one-third the risk of contracting the flu as a non-vaccinated person.

Unfortunately, using these numbers, we don’t know the absolute risk for anyone!

The group proportions are the absolute risks

We can estimate the absolute risk from the studies by looking at the proportion for each group in the Minitab output, and subtracting to calculate the absolute reduction. I’ll summarize this information below as percentages and even add in the results for two more flu seasons from another study that the CDC references (Bridges et al.3):

Flu season

Placebo

Flu Shot

% Point Reduction

1997/98

4.4

2.2

2.2

1998/99

10.0

1.0

9.0

2006/07

2.9

1.0

1.9

2007/08

10.8

3.4

7.4

Average

7.0

1.9

5.1

Notice how the risk of getting the flu varies by flu season? The differences are not surprising because the studies use different samples and the flu seasons have different influenza viruses.

So let’s look at the average of these four flu seasons. If you aren’t vaccinated, you have a 7.0% chance of getting the flu. However, if you do get the flu shot, your risk is about 1.9%, which is a reduction of 5.1 percentage points.

Hmm.  The "5.1% reduction" doesn’t sound nearly as impressive as the "67% effectiveness!" Both statistics are based on the same data, but I think the estimate of absolute risk is a more useful way to present the results.

Closing Thoughts about the Flu Shot Data

I was surprised by the results. While I knew flu shots were not perfect, I always got them because I thought they reduced my risk by more than what the CDC recommended studies actually show. Even if you aren’t vaccinated, your risk of getting the flu isn’t too high.

That probably explains why a number of people have told me that while they never get flu shots, they can’t remember having the flu!

These more subtle results made me wonder about flu vaccinations on a societal scale. Could the flu vaccine possibly reduce flu cases enough to save sufficient money (lost workdays, doctor and drug costs, etc) to pay for the vaccinations?

Bridges et al. conducted a cost-benefit analysis in their study. For the two flu seasons where they tracked flu vaccinations, infections, and expenditures, the vaccinations actually increase net societal costs. It would’ve been cheaper overall not to get vaccinated!

In light of this, I wasn’t surprised when I read an article on CNN.com that said, outside of the U.S. and Canada, other countries do not strongly encourage all of their citizens above 6 months to get the flu shot. According to the article, “global health experts say the data aren’t there yet to support this kind of vaccination policy, nor is there enough money.”

I understand this viewpoint better now.

However, I’m not trying to talk anyone out of getting a flu shot. I’m on the fence myself. While the risk of getting the flu in any given year is fairly small, if you regularly get the flu shot, you’ll probably spare yourself a week of misery at some point! You should always consult a medical professional to determine the best decision for your specific situation.

In another post, I look at the long-term benefits of flu vaccinations.

References

1. Beran J, Vesikari T, Wertzova V, Karvonen A, Honegr K, Lindblad N, Van Belle P, Peeters M, Innis BL, Devaster JM. Efficacy of inactivated split-virus influenza vaccine against culture-confirmed influenza in healthy adults: a prospective, randomized, placebo-controlled trial. J Infect Dis 2009;200(12):1861-9

2. Monto AS, Ohmit SE, Petrie JG, Johnson E, Truscon R, Teich E, Rotthoff J, Boulton M, Victor JC. Comparative efficacy of inactivated and live attenuated influenza vaccines. N Engl J Med. 2009;361(13):1260-7

3. Bridges CB, Thompson WW, Meltzer MI, Reeve GR, Talamonti WJ, Cox NJ, Lilac HA, Hall H, Klimov A, Fukuda K. Effectiveness and cost-benefit of influenza vaccination of healthy working adults: A randomized controlled trial. JAMA. 2000;284(13):1655-63