Online vs. Live Phone Poll Accuracy for 2016 Election State Polls

The last time I reviewed the accuracy of different survey methods for polling in the 2016 election, I found that live phone interviews came closer to the final national level result than online surveys did. As I’ll show below, it turns out that the opposite is the case when looking at polls for the state level.

Unlike in the last post when I just looked at polling error in terms of the margin of victory for Hillary Clinton at the national level, I take a more comprehensive approach to assessing polling error at the state level here. This entails looking beyond just error in candidate vote share margin to error in the vote shares themselves, for example. I’ll boil it down to a general takeaway at the end, but I take this comprehensive look to explore as many meaningful avenues for detecting polling error as I can, while leaving less room for doubt in the final results in the process. Here are the steps I took for this analysis:

1) For each state, I recorded the two average Clinton and Trump vote shares seen in the polls: the average among online surveys and the average among live phone surveys, thus making for four potential data points for each of the 50 states. Polls came from those listed on the HuffPost Pollster website for each state (an option under “Customize this chart” allowed me to narrow down polls to only “Internet” or “Live Phone” ones). I only included polls that were within two weeks of the election. That meant start field dates of the poll had to be no earlier than October 25th, and of course the end field dates were no later than November 7th. All 50 states had several online surveys during the final two-week window, but only 25 states had at least one live phone poll during this time frame. I’ll explain how I deal with this issue when comparing survey modes later. In a few states, the live phone polling average gets represented by only one poll in the final two weeks. This is a caveat worth keeping in mind, as greater error could result from averaging over a smaller set of live phone polls–or just one–compared to potentially greater accuracy from averages of a larger set of online polls.

2) Once I had Clinton and Trump polling vote shares in two different survey modes, I merged this state-level data with the actual state level Clinton and Trump vote shares from the election, taken from the U.S. Election Atlas.

3) From here, I was able to calculate error among online and live phone interviews for each state. Polling error is generally defined as the deviation of polls from the actual results. There are five different specific ways I gauge error.

a) Unadjusted margin: I first computed the Clinton margin of victory–which could be positive or negative–over Trump in each state to represent “actual margin.” I then calculated Clinton margins in polls conducted online–“online polling margin”–and in polls conducted through live phone interviews–“live phone polling margin.” (Note that the margin of victory measure can either be from Clinton’s or Trump’s perspective, as it produces the same error result as long as this stays consistent.) To get error values, I used the following formulas:

• Online polling error = | (actual margin) – (online polling margin) |
• Live phone polling error = | (actual margin) – (live phone polling margin) |

I include the term “unadjusted” because I don’t change anything about the vote shares from which margins are computed, such as making alterations in the denominator. That changes in the following error metric.

b) Two-party margin: For this metric, calculations for margins stayed the same as in Unadjusted margin except for one key change: Clinton and Trump vote in the actual election/online polls/live phone polls were calculated as two-party vote shares. In other words, third party and other selections were removed from the denominator. For example, here were the three adjustments I made in Clinton’s case:

• Adjusted Clinton share = Clinton actual vote / (Clinton actual vote + Trump actual vote)
• Adjusted Clinton online polling share = Clinton online share / (Clinton online share + Trump online share)
• Adjusted Clinton live phone polling share = Clinton live phone share / (Clinton live phone share + Trump live phone share)

I did the same adjustments for Trump’s three vote shares. I then created three adjusted (two-party) margins:

• Adjusted actual margin = adjusted Clinton actual share – adjusted actual Trump share
• Adjusted online margin = adjusted Clinton online share – adjusted online Trump share
• Adjusted live phone margin = adjusted Clinton live phone share – adjusted live phone Trump share

Finally, from here I could compute polling errors by survey mode:

• Adjusted online polling error = | (adjusted actual margin) – (adjusted online margin) |
• Adjusted live phone polling error = | (adjusted actual margin) – (adjusted live phone margin) |

c) Unadjusted Clinton share: In addition to evaluating error in terms of polling margin, I also did so with respect to vote shares for Clinton–and then for Trump–found in polls. Rather than examining the margin separating the two candidates in states, this metric and the next two metrics looks at whether polls properly gauged support levels for the major candidates–and whether online or live phone polls did a better job of this. The formulas for the unadjusted share metrics are very simple. Here is the metric for Clinton:

• Online polling error = | (actual Clinton vote share) – (Clinton vote share in online polls) |
• Live phone polling error = | (actual Clinton vote share) – (Clinton vote share in live phone polls) |

d) Unadjusted Trump share:

I took the same approach for the metric for Trump’s level of support in elections and in polls:

• Online polling error = | (actual Trump vote share) – (Trump vote share in online polls) |
• Live phone polling error = | (actual Trump vote share) – (Trump vote share in live phone polls) |

e) Two-party share: As was the case with Two-party margin, the adjustment for vote share entails looking only at the major party vote. For example, here’s how the calculation looked for Clinton’s vote share:

• Adjusted Clinton share = Clinton actual vote / (Clinton actual vote + Trump actual vote)
• Adjusted Clinton online polling share = Clinton online share / (Clinton online share + Trump online share)
• Adjusted Clinton live phone polling share = Clinton live phone share / (Clinton live phone share + Trump live phone share)

The above three steps were the same as the first free for two-party margin. For two-party share, I went straight to the survey mode error computation after these first three steps:

• Adjusted online polling error = | (adjusted actual Clinton share) – (adjusted Clinton online polling share) |
• Adjusted live phone polling error = | (adjusted actual Clinton share) – (adjusted Clinton live phone polling share) |

Absolute error turns out being the same whether it’s the two-party adjusted vote share of Clinton or Trump, so I don’t have to repeat this process from Trump’s perspective. That’s why I’ll only call it Two-party share.

Note: I include two-party vote adjustments in addition to unadjusted metrics for polling error because it’s been used to evaluate polling error in the past. In particular, it’s important for errors in vote shares (not margins) as polls could undercount support in candidate polling vote share but still have a good sense of the race when you just look at major party vote. However, I still view the unadjusted metrics as more meaningful, as underestimating support–even when differences in support are correct–still qualifies as error.

---

Now that I’ve thoroughly explained the process of calculation of my five measures for polling error, I’ll move into the results. The below chart sums up everything I’ve found regarding polling error by mode. The error values are broken up by 1) sample of states contained, 2) polling error metric (of the five different kinds described above), and 3) survey mode.

Data sources: HuffPost Pollster, U.S. Election Atlas.

In the upper half of the table above, I compare the mean absolute errors among only the 25 states that had both at least one online and one live phone poll in the final two weeks of the campaign. This is done for a fairer and more direct comparison (e.g. perhaps the 25 states without live phone polls during this window were harder to poll, and increased error for online polls while not doing the same for live phone polls). In the bottom half of the table however, I expand the mean absolute error for online polls to all 50 states, while keeping the live phone average at the 25 states that included this type of survey in the last two weeks.

Analysis:

There are 10 different points of comparison here for online and live phone polls. In seven of them, online polls have a smaller error and therefore were more accurate. In the most fairest level of comparison–using only the 25 states where both modes can be tested–online polls perform better than live phone polls across all five metrics.

For example in gauging the unadjusted margin between Clinton and Trump (comparing only across the 25 states), online polls erred by 3.87 points, but live phone polls were further off the mark with a 4.53 error. In measuring support for Clinton (in the unadjusted measure), online polls had a 2.01 mean error at the state level, while live phone polls had a 3.07 mean error. In terms of unadjusted Trump support, online polls (3.26 error) were also more accurate than live phone polls (4.38 error).

The bottom half of the chart shows that expanding to all 50 states to evaluate online polls results in online polls having greater error in three of the five metrics than live phone polls. But because live phone surveys weren’t included for these 25 other states, it does not really make for a fair comparison to include these states in the online poll error averages.

Key takeaway:

At the state level, polling was generally more accurate when conducted through online interviews than when conducted through live phone interviews. The differences in error weren’t drastic by any means, but are nevertheless consistent and strong across the board.

Caveat:

One important thing to point out in this finding is that I’m of course not making a causal claim about whether one type of survey administration reduced error more than another. To do that, I would have to control for in-house effects for different pollsters (i.e. accounting for the direction pollsters are typically biased), registered vs. likely voter populations, and other factors. Nevertheless, comparing the overall averages for online and live phone surveys and demonstrating error is still valuable in getting an overall picture of how close different survey modes got to the final result. That’s especially the case given that online polls are innovative and much less tested methods, and that live phone polls have the better track record of accuracy.

---

Just to better understand these results and visualize them in a way that includes state-specific survey mode comparisons (i.e. not just averages across all states), I’ll include a few graphs quantifying polling error by mode and state. I’ll use the three unadjusted error metrics from above. In all of these graphs, the closer the state abbreviations are to the x-axis of zero, the less error they had and thus the more accurate they were.

Unadjusted margin

On average, the error for this metric was greater among live phone polls than among online polls, but that wasn’t consistently the case for each state. For example, online polls were more accurate in Colorado, Georgia, Nevada, Utah, and Wisconsin, while barely outperforming live phone polls in North Carolina and Pennsylvania. Live phone polls outperformed online ones in New Hampshire and Ohio, and slightly more in Florida and Virginia. There was almost no difference in Arizona and Iowa.

Unadjusted Clinton share

For error in terms of estimating Clinton share of support, the states where online and live phone polls were more accurate were similar to those for the margin error metric. The only notable switches (in competitive states) were in Georgia, where live phone polls were more accurate, and in New Hampshire, where online polls were more accurate.

Unadjusted Trump share

Finally, there aren’t a lot of differences between state-level error in Trump support, seen in the above graph, and in Clinton support, seen in the previous graph. One thing worth noticing is the difference for the state of Wisconsin, the election result of which represented one of the bigger surprises. While error wasn’t that much different by survey mode for Clinton support, online polls were several points more accurate than live phone polls were for estimating Trump vote share in the state.

 

Advertisement