This is why American Polls got it wrong again: During the 2020 election, we took particular care to track two set of polls, one that showed Biden ahead and the other that showed Trump ahead. We were unable to understand why the gap between various polls was so high, it did not make sense at all. Today, we show you, why the polls were so wrong this time as well. See this tweet on October 20th, 2020

To understand the problem, we do a very simple comparison between Polls that showed Trump ahead with those that showed Biden ahead and see if there are some trends. We analyse two battleground states - Pennsylvania and Michigan. We analyse 3 pollsters in total

Pennsylvania

We compare the Emerson Poll that showed Biden ahead by 4 with the Trafalgar Poll that showed Trump ahead by 2. Biden is currently leading by 1 point in the counting which means that both the polls missed the final result by about 3 points. 

Key Findings

• Amongst White voters, Trafalgar shows Trump ahead by 11 points while Emerson shows Trump ahead by 10 points
• Second, amongst Non-White voters, both the polls show Biden ahead by 52 points
• When it comes to female voters, Trafalgar reported a gap of 8 points, Emerson too reported a gap of 8 points
• However, amongst men, Trafalgar showed a lead of 10.8 points while Emerson showed a lead of just 1.6 points. That was the difference.

Emerson Polls follow a mixed mode data collection while Trafalgar used an IVR method. In terms of the age weights, each of the polls used different weights for those older than 64 year of age. It was 23% for Trafalgar and 25% for Emerson

Michigan

Biden is ahead by 2.7% at the moment in the counting. Trafalgar poll predicted that Trump will win by 2 points while an Insider advantage poll predicted that Biden will win by 2 points (Closer to the result)

The Findings

• Amongst white voters, Trafalgar showed Trump ahead by 10 points, the Insider Advantage Poll shows Trump ahead by just 3.6 points
• Amongst non-white voters, Trafalgar showed Biden ahead by 36 points, the Insider Advantage Poll shows Biden ahead by 28 points
• Amongst women, the Trafalgar poll shows Biden ahead by 3.1 points, the Insider Advantage Poll shows Biden ahead by 6.4 points
• Amongst men, the Trafalgar poll shows Trump ahead by 8.9 points, Insider advantage shows Trump ahead by 2.4 points

Interestingly, both Insider Advantage and Trafalgar reported very similar results in Pennsylvania and very different results in Michigan. Insider advantage also uses a mixed methodology for data collection, similar to Emerson. But there is a huge variance in the results amongst male voters with Trafalgar showing large leads while Mixed Method Pollsters showing lower leads. In terms of weights, in the Insider advantage poll, voters older than 64 made up for 21.8% of voters, in case of Trafalgar it was 23.1%

Consistent trends across the 3 pollsters are 1. Trafalgar (IVR Method) shows big leads amongst White voters and Men, 2. The Mixed Methodology Pollsters like Emerson and Insider Adavntage show much lower gaps amongst Men but it is not necessarily always producing accurate outcomes, 3. Age weights are not the same across all the pollsters

Looking at all the data, here are our major takeaways

• Underneath, polls produce similar results in some cases and dissimilar results in other cases. Mixed methodologies appear to be producing lower gaps amongst Male voters but it is unclear what is an effective mix, What % of interviews should be IVR and what should be web or text. Pollsters are yet to crack this which is why the same pollster is ending up with different outcomes across the States. In case of Pennsylvania, Emerson missed the final results but Insider Advantage did not in case of Michigan using the same method
• Turnout weights: Each Pollster is using different turnout weights. What this means is they don't have an accurate idea as to which voting blocks will vote more and which groups will vote less in an election. Using stated intention questions to guage enthusiasm is not turning out to be precise. This is not an easy problem to fix and without this getting fixed, the results will continue to miss the target. 

Both these issues exist in every opinion poll around the world whether it is UK, Israel, India or even the recently concluded New Zealand Elections. It is therefore recommended that we prepare ourselves for wider margin of errors than stated by pollsters themselves. Summary numbers like the ones produced by RCP or Nate Silver are actually more reliable than individual polls.