Why pollsters and pundits got the election wrong

Hillary Clinton was not the only loser on election day. Scores of political experts and pundits also got it wrong by predicting that she would win. What went wrong?

For an answer, I interviewed Mark Gray, a polling expert at the Center for Applied Research in the Apostolate, a Catholic research institute located at Georgetown University.

Gray defended the national polls, which accurately predicted that Clinton would win the popular vote.

“I see these reports about the biggest failure ever,” he said, “but on that national level, it’s in line with previous elections. Given the margin of error and how close the election was, the polls aren’t off at that level.”

Pollsters only survey a sample of the population and therefore cannot be expected to be perfectly accurate all the time. This is often expressed by saying that the margin of error is a certain number of percentage points, say 2 or 3 percent. The margin of error is supposed to measure the maximum amount by which the sample results are expected to differ from those of the actual population.

Depending on how you aggregate the polls, the difference between the national popular vote and the final poll aggregation was 1, 2 or 3 percent, which was within the margin of error. “They’re not far off,” Gray said.

He agreed with Nate Silver, the nation’s most famous number cruncher at fivethirtyeight.com, that the final polls were about as accurate as they were four years ago, except that year President Barack Obama’s win was bigger than predicted.

Gray noted that every election since 2000 has been close.

“We’ve just got another contest where the margin of victory is so small that the margin of error is bigger for our polls and our surveys,” he said. “So on that level, I don’t see a failure.”

In the United States, presidential elections are not determined by the national popular vote but by the electoral college, which means that the votes in individual states matter and so do their polls.

“As I was looking at places like Pennsylvania and Michigan, the polls there did tighten in the final days to where it looked competitive” he reported. “But the one case where clearly the polls were wrong is Wisconsin. Trump never led there. In fact, most polls had Hilary Clinton ahead by 6 percentage points.”

But in most other close states, “it’s a case where the margin of victory and the margin of error were relatively close,” Gray said. “You’ve got just close election.”

On the day of the election, Gray told his students that Trump was going to win Florida and North Carolina.

“I initially was going to predict that Florida and North Carolina were going to be Clinton wins,” he explained. “But then I saw a news story that she canceled her fireworks” for her victory celebration in New York City. Gray thought, “You only cancel your fireworks if you know your polls are saying, ‘You might not win Florida.’ ”

He felt that “North Carolina was even a lesser bet for them than Florida was, so in my prediction, I gave Florida and North Carolina to Trump.”

“Maybe that’s magical thinking, not based on polls,” he admitted by reading that into their cancelling the fireworks. But “the candidates always have better polling data than we do or the media does. My assumption was okay, they’re unsure about this.”

“The three states I got wrong were Pennsylvania, Michigan, and Wisconsin, which I didn’t think were likely to be areas that Trump could win,” he confessed. “I should have known Pennsylvania.”

Gray recounted how a couple of months before the election, he and his family drove through Pennsylvania. “There were Trump signs everywhere,” he reported. He said to his kids, “The first Hillary sign you see there’s a $100 prize.” They only saw one.

“We were driving all around Western Pennsylvania, and it was just covered with Trump signs. These were rural areas, granted, but I think a lot of people in the media don’t get out into those areas and see the energy.”

“I imagine the same kind of thing was evident in places like Michigan and Wisconsin,” he continued. “This rural vote was to some extent something that people didn’t necessarily think about. It wasn’t necessarily captured accurately in the polls.”

Nor does Gray think the role of third parties was insignificant in some of the states that were close.

“I’ve seen these figures in Florida and some of these states,” he said. “If Jill Stein isn’t there and those voters vote for Clinton, and maybe some of the Libertarians as well, the election is completely different.” Some may have voted third party as a protest vote; others may have thought Clinton was going to win so it did not matter. Or it may have been a movement they believed in. Whatever the case, it made a difference.

But the problem was accentuated when people took the polls and used them to predict the likelihood of Clinton winning. For example, the Huffington Post had her probability of winning at 98 percent.

Gray noted that Silver admitted that his predictions were wrong during the primaries because he was acting like a pundit rather than a statistician. Silver didn’t believe that Trump could win. “By the time of the general election Silver had come around, had made the correction, and he was being criticized for his Hilary likelihood of winning percentage not being high enough,” Gray recalled.

Silver gave Clinton a 71 percent chance of winning, which means Trump had an almost one-out-of-three chance of winning. Others put the probability of Clinton winning at 85 percent (New York Times), 89 percent (Predict Wise), 92 percent (DailyKos), 98 percent (Huffington Post) and less than 99 percent (Princeton Election Consortium). Betting sites gave Trump an 18 percent chance of winning the night before the election.

“The New York Times, the Huffington Post — all these other outlets that thought they had their own secret recipe to predict the election — were wrong,” said Gray. “They’ll blame the polls, but the polls did not show a 98 percent likely win. It just doesn’t work that way. These models include assumptions that take that polling data and transform it into something that I think might have been wishful thinking.”

One of the most important assumptions in predicting elections is voter turnout.

Gray admitted predicting turnout of voters is very difficult. “Is this person I’m interviewing really going to show up or not?” asks the pollster. Answering that question, according to Gray, “becomes more magical thinking than science.”

“The numbers I’ve seen on turnout [this year] show that the Republicans have a pretty stable turnout and that Democrats underperformed” with many groups, reported Gray. “Trump voters were more energized than Clinton voters,” as he saw driving through rural Pennsylvania. “This rural vote was to some extent something that people didn’t necessarily think about. It wasn’t necessarily captured accurately in the polls.”

And of the people who did turn out, Clinton underperformed in comparison with Obama with some groups whom the pundits expected her to do better.

One of the surprising results of the exit polls was that 29 percent of Hispanics voted for Trump, a higher percentage than they gave Romney (27 percent), when everyone was predicting a tsunami of Hispanics voting for Clinton.

“The Hispanic vote isn’t monolithic,” especially in Florida, Gray explained. “The Cuban vote historically leans right and isn’t necessarily concerned with issues of immigration.” As refugees or children of second, third generation of refugees, they have no fear of being deported. Likewise, “If you’re from Puerto Rico, you’re a U.S. citizen” and deportation is not an issue.

Most pollsters agree that Hispanics turned out more this year than four years ago; the exit poll showed them making up 11 percent of all voters this year as opposed to 10 percent in the last presidential election. But Latino Decisions, a polling firm that focuses on the U.S. Latino population, believes that the exit poll underestimated the percent of Latinos voting for Clinton. In examining the dispute, FiveThirtyEight sides with the exit polls. “No one understands the Hispanic vote better than Matt Barreto,” the founder of Latino Decisions, said Gray. “I'd agree with his data and tracking.”

Pundits were also surprised that Clinton did not get a larger percentage of the women’s vote. Gray believes that women no longer doubt that they will see a woman elected president in their lifetime, so the urgency is not there as it was for Catholics in 1960 or African-Americans in 2008.

Gray noted that the exit polls do have problems which make their results suspect.

“I’ve always told my students, the worst form of political polling is the exit polls,” said Gray. “Your average person who does the exit polls might have a day of telephone training; polling isn’t their fulltime job and they don’t have a background in interviewing. They’re told to interview every seventh person that comes out. When they’re observed, we don’t see a lot of regimented following of those rules.”

Although they do have a larger sample than most polls, Gray believes, “it’s always been an imperfect and fuzzy view.” And “since we have so much early voting, I don’t know how good of a window exit polls are into the results.”

Did the letter from FBI Director James Comey announcing a reopening of the investigation into Clinton’s emails make a difference?

“There was definitely a shift in the polls,” said Gray. “There are things that happen in the last week of an election that can affect those undecided one way or another.” He pointed to 2012 superstorm Sandy and its aftermath, including Obama’s visit to the storm site and the hug from New Jersey Gov. Chris Christie. “Obama’s lead was not as strong [in the polls] as it would be in the end,” he recalled.

I, of course, had to ask Gray about the Catholic vote, something that he has watched for years.

“The Catholic vote is just a mystery to me,” he confessed. “You’ve just got a lot of volatility up to election day.”

“I wouldn’t expect it to move up and down within that one period in the way that the polls were showing,” he admitted, although Catholics do switch back and forth from election to election.

Some of the volatility could have been due to the margin of error in the polls when looking at a subsample like Catholics. The smaller the subgroup, the greater the margin of error. 

“Most polls are like 1,000 voters, so they’re getting 220, 230 Catholics,” he explained. “So you’ve got such a large margin of error with that subgroup that I expect there’s going to be more volatility than there is in the actual election.” This is even truer of a smaller subgroup like white Catholics.

The exit polls start with a larger sample, which would lead to better results with subgroups. “If there weren’t all these other issues with the exit polls, you would have more confidence in their numbers,” Gray said.

“I don’t think we have an explanation yet of why Catholics voted the way they did even if they did vote the way the exit polls said they did,” said Gray. “If you would have asked me before the election, I would have said the most recent polls show that Hillary Clinton has a pretty strong lead among Catholics” but according to the exit poll, Trump won 52 percent of the Catholic vote.

Gray is waiting to see results "from Gallup and the American National Election Study before I’m convinced of the precise percentages.” On CARA’s website, Gray has published the numbers on the Catholic vote from 1952 through 2012, which show that different polls show different numbers for the Catholic vote in presidential elections. In some elections, the poll numbers for the Catholic vote differed by up to seven percentage points among the polls. When he has numbers from these sources, he will update this page.

Besides pollster and pundits, political scientists also proved to be wrong about the election and the strategy that they had recommended to Clinton, according to Gray.

“Political science in the last 15, 20 years has considered themselves the engineer of this new mobilization model of knocking on doors and making these personal appeals and thinking that’s the magic to getting turnout,” he explained. The data mobilization strategy required getting all your supporters in a database and then contacting them frequently to make sure they voted.

“The Democrats and Hillary had that machine working in this election. Trump had nothing. He had television and rallies. He had no organized data mobilization. The Republicans are famous for having an almost comically bad turnout machine versus the Democrats.”

And yet Trump won. “It’s a big blow,” Gray thinks, to the idea that data mobilization works. “Personally, I think the machine may have never been what it was. It’s someone like Barack Obama that can mobilize voters like that.” Clinton did not have the same appeal Obama did, and the data mobilization strategy did not overcome that lack of appeal.

On the other hand, “Trump didn’t need a machine. All these signs in people’s yards in rural America, they were going to be there [voting] whether someone knocked on their door or not.”

What does the future hold for American politics?

“If things play out as they have historically, the Republicans will have difficulty holding on to the Senate,” said Gray. Parties historically get one or two terms in the presidency, with Bush following Reagan an exception. “This election, again, showed that’s a very difficult thing to do,” he said. “These first four years will be critical to knowing if Trump gets re-elected or if you have Democrats winning back in four years. It will be very difficult, I think, for the Republicans to hold on to the Senate just given the way history works.”

“There’s a realignment going on,” explained Gray. “If you look at the county level map, the Midwest looks like fertile ground for Republicans in the future. The Sun Belt looks like fertile ground for the Democrats in the future. Texas, Georgia, Florida — those states are going to go purple and they’re going to go blue. You might see a very different map over the course of the next two or three presidential elections, a political realignment, which people have been thinking about since the 1960s.”

Gray also noted the growing number of independents. “Just like religious affiliation where you have this growing number of people who don’t affiliate with a particular religious affiliation, you have a growing number of people who don’t affiliate with the traditional parties. This is proportionally [larger] among younger people, among Millennials who don’t necessarily think in the same terms of membership and identity in the way that older generations do.”

“In this election, that group did not break the way that Hillary Clinton needed them to,” he said.

When I suggested that the Democrats might win state legislative elections in time for redistricting after the 2020 census, he cautioned me not to blame gerrymandering for the Republican majority in the House of Representatives. “It’s not the real problem,” he said. “There’s some gerrymandering, but the Democrats will not take the House back until you get some liberals living on both coasts and in urban and suburban areas to move to Kansas and Nebraska and Iowa. I don’t think they want to do that.”

The problem for the Democrats is that “the House is geographic,” he said, just as “the Electoral College is geographic. So, in some sense, the total population doesn’t matter, it’s the distribution of that population.” In a huge part of the country, “the “fly over states” as people call them, things aren’t changing demographically, economically. That disaffected area in the end, I think, is the story of this election.”

[Jesuit Fr. Thomas Reese is a senior analyst for NCR and author of Inside the Vatican: The Politics and Organization of the Catholic Church. His email address is [email protected].]

Editor's note: We can send you an email alert every time Thomas Reese's column, Faith and Justice, is posted. Go to this page and follow directions: Email alert sign-up.