Vice President Kamala Harris was polling at +1.2 points in Michigan and +1 points both in Wisconsin and nationally.
President-elect Donald J. Trump won Michigan by 1.4 points, Wisconsin by 1.1 Points, and the national popular vote by 2 points. He was the first Republican to win the national popular vote since George W. Bush in 2004.
This leaves polling errors of 2.6 points, 2.1 points and 3 points, respectively. The average miss of the swing state and national polling average was 2.2 points. The highest error was 3.4 points (Nevada), and the lowest error was “only” 0.9 points (Pennsylvania).
What happened? What did all of these polls miss? And how can they fix it?
Let’s take a step back. At its simplest, polling is asking registered or likely voters what candidates they plan on voting for to get a feeling of where the election will go. Hundreds of firms specialize in this, and they all have their own methodologies and produce different results. The common strategies for polling are phone polling and online polls. Pollsters then take the percentage of respondents who said they supported a specific candidate and find a net percentage to figure out how many percentage points a candidate is ahead by. This is the polling average. It allows us to form an image of how well the candidates are doing and where they have room to improve before the election even happens.
Polling averages are more than just the average of the polls though. The polling averages that are most commonly used and referred to weigh historically reliable polls more, incorporate state polls into national polling, and adjust the polls for “house effects” a term referring to some polling firms’ tendency to favor one candidate over another.
Except for a few notable pollsters, current polling averages are riddled with errors. This hasn’t always been the case. In 2004, polls averaged Bush +1.5, and the election ended up going to him with a popular vote of Bush +2.4. In 2008, polling averaged Obama +7.6, and he ended up winning the election and the popular vote by 7.3 points. The polls started losing accuracy in 2012, where the polling average was Obama +0.7. He still won the election and the popular vote, but he won them by 3.9 points.
Then there was 2016.
Polls ended up being decently accurate to the popular vote, predicting Clinton +3.9 with the final popular vote being Clinton +2.1. However, the Electoral College differed significantly from the popular vote. This almost two point difference resulted in Trump walking away with a total of 304 electoral votes. Gaining those two points would have helped Clinton win Michigan, Wisconsin, Pennsylvania and Florida, the states that decided the election.
Forecasting, directly putting a percentage on how likely people think an event is, and is much harder to judge the accuracy of, added fuel to this fire by further minimizing the perceived probability of Trump winning.
In 2020, the polls had Biden ahead +8.4 points. The final result was Biden +4.5 points. This was the second major underestimation of Trump, a polling error of 3.9 points.
Where does this leave pollsters? They were pretty spot on in 2008, decently accurate in 2004, and at least they got the right candidates in 2012 and 2020, even if the errors were pretty severe. 2016 was the hardest miss because the Electoral college winner didn’t win the popular vote. The polls had yet another significant miss this year.
Even though this is the third significant underestimation of Trump in a row, polling still has a future. People still want to know which candidate is ahead in a race.
However, we will likely see drastic changes in methodology such as focusing more on state to state polling. Pollsters have to work hard to regain the trust of voters after three straight misses. It’s not enough to just change methodology, numbers have to improve.
There is a clear difference in accuracy between pollsters who use more traditional methods such as calling-based polling, and those who use more modern methods such as web-based polling. This distinction is most clear when comparing historically accurate Ann Selzer in Iowa, who was once considered “The Best Pollster in Politics,” and AtlasIntel, a modern firm that was the most accurate pollster in 2020, the first American election they polled.
Shortly before the election, they offered two very different ideas of what to expect. Seltzer showed Harris +3 in Iowa while AtlasIntel had Trump +1.2 nationally.
Iowa ended up going 13 points for Trump, making Selzer’s error 16 points, while Trump won the popular vote by 2 points, making AtlasIntel’s error 0.8 points. All of AtlasIntel’s swing state polling had a similarly small error, significantly more accurate than the average pollster.
Pollsters will change their methodology because of this error. Here are a few things to expect.
The first is web-based polling. This polling method, predominantly used by the previously mentioned AtlasIntel (538’s only A ranked pollster to use this method) clearly has an effect on the accuracy of their polls, and it would not be surprising if many pollsters shifted towards web-based polling in an attempt to recreate this accuracy. Online polling likely reaches more of the population since many people don’t pick up their phone for numbers they don’t know.
The second is commonly referred to as “neighbor polls”. These are polls that ask respondents who they think their neighbors will vote for instead of who they are voting for. A study by Columbia University analyzed this, stating that the publicly available polls using this methodology, “showed Harris’s support was several percentage points lower when respondents were asked who their neighbors would vote for, compared with the result that came from directly asking which candidate they supported”.
Polling is not going anywhere. However, pollsters must make several substantial changes in order to regain people’s trust and stay afloat in this political environment.
Note: I use here polling averages from Nate Silver’s website, Silver Bulletin and election results from the New Yok Time’s Presidential Election Results Map.
