Rewritten on 1 September 2021
Some Vaccine Effectiveness Studies can give misleading results especially those having shorter timeframes and those using real World Data. This article will use real World data to illustrate why.
Reasons to Carry out Studies
It is not uncommon for countries to carry out vaccine effectiveness studies. They might have carried out the studies under their peers' or political pressure. They might want to find out why there are more people infected. Traditional and other accurate studies could cost a lot of money.
What's Vaccine Effectiveness and Efficacy?
Both are measuring how well the vaccine will work. The main difference is in the way of measurement. Vaccine Efficacy uses clinical trial data, whereas Vaccine Effectiveness uses real-World data. The other differences are as shown in this table.
| | Vaccine Efficacy | Vaccine Effectiveness |
| Range | Selective group according to age, sex, ethnic & known medical condition | Real-World data is where is |
| Sample size | At least 30,000 selected participants are divided into 2 groups; the vaccine and Placebo group | The community at large. No grouping. Large sample size. |
| Controls | The participants are not told about their vaccination status. Whether they were vaccinated using a vaccine or just plain salient water. Participants are asked to go back to the community and work as per normal routine. | Members in the community are well aware if they were vaccinated. The unvaccinated and those having only 1 dose will usually take extra effort & precautions to protect themselves. The government often introduce infection control measures |
| Testings | The participants are tested at regular intervals. | Testings are done on an ad-hoc basis |
| Exposures | The total and actual numbers of vaccinated and unvaccinated exposed to Covid is known | The total and actual numbers of vaccinated and unvaccinated cannot be accurately determined. |
| 95% Confident Interval (CI) of Outcome | Usually narrow, not more than 10% except when the sample size is too small | Usually wide, can exceed 50% and sometimes can go negative. . |
| Outcome | Less biased because participants do not usually know what “vaccine” they have taken | Member unvaccinated or partly vaccinated will exercise extra precautions to control their activities. |
Are there no other means to accurately determine Vaccine Effectiveness?
Yes. There are many other ways and means. Some of the common ones are listed as follows:
These study methods are similar to those Clinical Vaccine Trial tests used to determine Vaccine Efficacy. In fact, these other methods are variants. They claimed to be fairer and more accurate than the effectiveness measurement. Many of these studies are using selective candidates and not using real World data.
But they would not carry out these studies if the existing vaccine is still effective. This is mainly because it is not quite ethical to "bluff" those participants who have taken salient water instead of the vaccines. It is also expensive.
One other way is to do scheduled household visits. Every member of the household will be tested for infection at regular intervals. This will reduce the errors caused by not knowing how many had been exposed to the risk of infection. However, the result could still be biased because those unvaccinated are aware that they have not been vaccinated. They will take extra preventive precautions.
What are the most common problems in Measuring Vaccine Effectiveness?
We often use real World data & the standard efficacy formula for effectiveness calculation. The accuracy of the results can be a problem.
The results can contain lots of unknowns & variables. What we are measuring is not the vaccine effectiveness. It could be the effectiveness of the vaccine plus the other control measures. The results could also be biased.
The standard formula compares the probability of the vaccinated against unvaccinated people. The formula does not work when the majority of the population has been vaccinated. An example is when the unvaccinated people approaching 0%.
Examples?
Singapore’s Ministry of Health (MOH) recently changed its reporting method. They used to tell the public about where and who were infected. Now, they concentrated on telling people about what will happen to vaccinated and unvaccinated.
The purpose is to encourage people to take vaccines. The MOH website presented 2 interesting charts. One showing the hospitalization of those infected (Figure 6). The other is showing the number of infections (Figure 10). The former is showing in a 28-day interval starting from 29 June 2021. the latter in a 14-day interval starting from 20 July 2021.
The charts give the following information
a) How many of the infected people are vaccinated and unvaccinated?;
b) How many of them are infected and how many are hospitalized?
c) How many are seriously ill requiring oxygen support, ICU bed and how many have died?;
Using the real World data collected from MOH, an attempt was made earlier using the standard formula of attack rates to work out the Vaccine Effectiveness but the results were quite a blizzard and confusing. These are as shown in the following examples.
Chart 1: % of Population vaccinated with 1 dose and full 2 doses
Chart 2: The Virus Attack Rates by Vaccination Status
Chart 3: Vaccine Effectiveness Calculation
Explanations?
Chart 1: This chart shows the vaccination status of Singapore residents over the period specified. The working example in this article will use the vaccination status as the base to work out the attack rates. It will assume entire Singapore is one community Chart 2: This chart shows the attack rates of vaccinated and the unvaccinated. The data is extracted from Figure 6: Local Cases in the Last 28 Days by Vaccination Status & Severity of Condition. The attack rate is explained in this wiki. Chart 3: This chart works out the vaccine effectiveness using the vaccination status as shown in Chart 1. The effectiveness is worked out based on the same formula in this wiki Result Interpretation?
1. The raw attack rate & the Effectiveness calculations appear to suggest that those unvaccinated has rate and effectiveness better than those vaccinated; this is not logical; 2. The effectiveness also went negative for those having 1-dose and also for those full-dose people recently; this is not possible.
The following could be the reasons: 1. The real World data are biased. The unvaccinated know that they are vulnerable. They have either hibernate or took extra precautions themselves from being caught by Covid. On the other hand, many vaccinated people will usually not follow the protocols & SOPs.
2. The attack rate of the full vaccinated has gone negative because the spreading of the disease had gone unnoticed for some time. By the time it was discovered, many people, especially the elderly, had already been infected. The attacks on vaccinated people went record high.
3. Almost all the elderly and those people in customer service have been vaccinated in Singapore. The elderly are most vulnerable to infection and the customer service people are constantly exposed to infections. Their chances of being infected are much higher than unvaccinated.
4. The Government's various control measures have distorted the attack rates.
In Conclusion
If we use real World Data and the standard formula to work out the effectiveness, we are not actually measuring the vaccine effectiveness but a combination of vaccine effectiveness as well as other measures to prevent Covid-19 infection. These measures are shutdowns & isolations, mask wearings, etc. Part 2 of this article will suggest a method where we can monitor the effectiveness of this combination effect.
