The big mistake polls are making in 2024

To avoid their big miss in 2020, pollsters are making their polls look more like 2020. Here's why that's (probably) a big mistake and why we should stop looking at polls entirely.

Oct 12, 2024

Welcome to the weekend edition of The Experiment, your official hopepunk newsletter. If you’d like to support my work, become a paid subscriber or check out the options below. But even if you don’t, this bugga free. Thanks for reading!

r/dataisbeautiful - a map of the united states with red squares

Hi, I’m Jason, and I’m a pollaholic. I usually delete Twitter/X from my phone after the regular baseball season or the Baltimore Orioles are eliminated from the playoffs, but I haven’t yet. I can’t. At first it was because Kamala’s flawless ascent made all the arrows point in the right directions, microdosing my hopeful brain with dopamine.

Now things have gotten a bit squiffy, and yet I scroll. I see cracks in the Blue Wall and shifts in the Sun Belt. Donald Trump’s closing message is playing Racist Chicken with America while Republicans accuse Joe Biden of sending hurricanes through Republican counties. It’s like trying to get a clear view of the gas chambers in a fun house mirror. What I’m seeing is so absurd and venal that it can’t be real; also, I’m not getting a clear view.

We’re told over and over that the presidential election is a coin flip, the winning margin residing within the pollsters’ margin of error. Instead of clarity of sight, polls offer us the fog of war made more confusing by the residual terror of 2016 and 2020 when the polls undercounted Trump’s vote. And now, with Nate Silver joining Polymarket, a predictions market backed by Peter Thiel’s venture fund, even Celebrity Nerds Who Can Tell The Future (to borrow Frank Spring’s apt phrase) can’t be trusted.

It doesn’t help that Republicans seem to be gaming the public polling industry and prediction markets to make it seem like Trump is doing better than he is. Over the last two weeks, Republican-aligned polls have flooded the market (against one Democratic-aligned poll), skewing polling averages in Trump’s favor. Other than pacifying He Who Shall Not Be Shamed, I can’t fathom1 why Republicans would want to make things look better than they are—as opposed to actually improving their performance with voters—unless they are setting the stage to claim the election was stolen. Again.

Put aside Republican shenanigans for a moment. There’s something that most non-partisan public pollsters are doing that is (probably) skewing the results. Actually, they are definitely skewing the results in a way that is possibly making them inaccurate across the entire industry.

And here’s the thing: They’re doing this for the best possible reason. In 2020, public polling undercounted Republicans by an average of 4.6% nationwide. In three states they undercounted Democrats, and only in Louisiana did they get it absolutely right. But on average, after blowing it in 2016, they blew it worse in 2020. So to prevent that from happening again, most of them adopted a solution that might mess things up even more this year.

What they’re doing in 2024 that is new is called “weighting on recall vote,” meaning they ask you who you are voting for this year and then who you voted for in 2020. To make sure they aren’t undercounting Trump’s vote, they give extra weight to the responses who said they voted for Trump in 2020. This makes the 2024 modeled sample more closely resemble the 2020 electorate. Basically, if you say you voted for Trump in 2020, your survey response counts for more (that is, given greater weight) than if you say you voted for Joe Biden in 2020.

“Weighting on prior reported Trump vote feels pretty close to weighting on current Trump vote.”

As pollster friend of mine says the intentions here are good. “The reason they were getting poor results before was underrepresenting more conservative GOP voters - which you could see looking at party and ideology,” he said. “This is a short hand fix, which is fine for a media poll.”

You, reading this now, are probably immediately thinking of why weighting on recall vote invites mayhem. Can people be trusted to remember accurately who they voted for in 2020 or even whether they voted? No, of course not. Memories are not recordings. They are creations, and people tend to say they voted for the winner of an election simply because brains are stupid.2 For this reason alone, weighting by recalled vote increases support in the poll for the party that lost the last election.

Weighting by recalled vote also recreates the 2020 electorate by assuming the 2024 turnout will be similar, which has the paradoxical effect of just making Trump voters worth more in public polls. “Weighting on prior reported Trump vote feels pretty close to weighting on current Trump vote,” said my pollster friend. If you can feel your brain breaking a little, good, we’re on the same page.

What public polls that weight on recall vote also will not pick up are changes since the 2020 election. Did Jan. 6 help a lot of Republicans see the light on Trump? Sure did. Did the Dobbs decision fundamentally reorder how women vote? Probably. Did the whackadoodle pandemic reactions radicalize a lot of formerly normal people? Unfortunately. Did pandemic migrations reorder the electorates of Arizona, Florida, and North Carolina? Absolutely.

We’re also seeing suburban and college-educated voters reseat themselves as Democrats amid a bigger cultural reshuffling that The Bulwark’s Marc Caputo recently identified:

Will public polling modeled on a 2020 electorate pick up these massive upheavals? Not so much.

And that brings me to the biggest reason I choose to be optimistic—and why we should all ignore the polls and go back to sharing videos about animals making unlikely friends. Before I explain the big reason, here’s a video about a tiger, a bear, and a lion.

In pre-vaccine 2020, Democrats thought it would be a good idea not to knock on doors and instead relied on texting to get out the vote; Republicans went ahead and knocked on doors. A 2000 study found that in-person canvassing could boost turnout by 9-10%, while methods like texting or digital outreach had significantly lower impacts. It’s unlikely that the Trump campaign, which faced its own headwinds, enjoyed the full 10% advantage over Biden. In fact, voter turnout was high across the board, but the fact that Republicans knocked on doors and Democrats didn’t has to account for at least some of what was perceived as the 5% polling error.

This year, we’re seeing the reverse. While Kamala has inspired thousands of Americans to volunteer to get out the vote (and raised millions to pay for the infrastructure to support this), Republicans have gone in another direction. This year, instead of funding get-out-the-vote programs, the RNC is focusing on election integrity and outsourcing GOTV to Turning Point USA, an extreme, pro-MAGA group, which is using an app to turn out low-propensity swing voters.

“Their strategy is bad. They know how to talk MAGA, they know how to message the base,” said Tyler Montague, a Republican strategist from Arizona and a longtime Turning Point critic, referring to the former president’s Make America Great Again movement. “But they literally don’t know what to say to a swing voter. They alienate these people.”

This is going about as well as expected. Anecdotally, doors are not getting knocked for Republicans in swing districts. “I’m very worried about the get-out-the-vote effort on the Republican side,” a pro-Trump GOP strategist told The Hill. “The legal challenge part of it, Republicans have been light-years better than 2020. Where I think I’m a little bit worried is the get-out-the-vote.”

And on the Democratic side, those whose business it is to track voting patterns are encouraged about what they’re seeing so far.

“Hate to rain on the ‘Democratic freak out’ parade,” posted MSNBC analyst and University of Miami professor Fernand R. Amandi, “but if you’re curious about data of voters actually voting in this election, preliminary early voting numbers in Michigan, Pennsylvania + Wisconsin still look VERY good, so far, for Kamala Harris — better than Biden 2020 numbers.”

Tom Bonier broke down the early vote in Pennsylvania and noticed a spike in women voting early with “the biggest increases among women of color, especially Black women whose turnout is 248% of their turnout at this point in 2020, compared to 146% for white women.” Patrick Schuh of America Votes broke down the early vote in Michigan and figures that “Dems could enter EDay with more than 1.2-1.4m votes banked.”

And Rosenberg, whose Hopium Chronicles newsletter is becoming a must-read, says Democrats are similarly winning the early vote in the seven swing states, banking millions of votes and freeing up Democratic campaigns to focus get-out-the-vote efforts on stragglers, infrequent voters, and special snowflakes who need their hands held while they’re being driven to the polling place. “We are winning the election but have not won it yet,” wrote Rosenberg. “In sum - I would much rather be us than them.”

The public polling modeled on a 2020 turnout won’t pick this up because it’s designed not to. So as Americans head to the polls, I’m backing away from the polling. I might even mute variations of the word “polling” on Twitter and limit my exposure to the one true Nate, Nate Cohn of The Tilt. Instead, I’ll focus my time on sending letters to voters to do my small part to get out the vote.

Hi, I’m Jason, and I’m a GOTV-aholic.

An earlier version misidentified Marc Caputo’s employer as Politico, which apparently was three jobs ago.

Jason Stanford is a co-author of NYT-best selling Forget the Alamo: The Rise and Fall of an American Myth. His bylines have appeared in the Washington Post, Time, and Texas Monthly, among others. Follow him on Threads at @jasonstanford, or email him at jason31170@gmail.com.