Revisiting The Live Interview VS. Online/Automated Poll Split And What It Means For The 2016 GOP Primary

As Donald Trump continued to maintain a decisive polling advantage towards the end of 2015, a noticeable split in what different kinds of polls captured was popularized.

At one point in national polls, Trump did 5.7 percentage points better in online polls and 5.9 points better in automated phone polls than he did in live interview polls. Similar splits occurred in polling of the Iowa and New Hampshire races. Conducted over the phone but with an actual person on the other end, this latter type of survey has marked the traditional form of measurement in pubic opinion research. As of late, however, and especially during this 2016 election cycle, non-live interview surveys have begun to spring up more frequently, a result stemming from both practicality and cost-saving efforts.

So when this divide between live interview and online/automated results arose, a new source of doubt was cast over the strength of Trump’s candidacy and polling position. More traditional survey styles have been shown to more accurately predict eventual vote share. Notably, with internet polling, there is always the risk of picking up more people who have a lesser likelihood of voting, and thus (wrongly) overrepresenting their opinion in survey results.

In the 2012 general election and 2014 midterms in the United States, as well as 2015 elections in the United Kingdom, non-live interview polls have surged in predictive power. Nevertheless, they remain untested in primaries, where party identification becomes less of a guiding factor than in a general election setting. In the 2012 GOP primary, challengers–who ultimately faded out as part of the “discovery, scrutiny, decline” cycle–to Mitt Romney’s lead fared much better in online surveys, only to see their position collapse after some time.

Yet just as this phenomenon started to gain traction, the divide in Trump support closed. Perhaps in some way due to his Muslim ban pronouncement in early December that garnered even more media attention and consequently more support all-around, the gap between live interview and non-live interview polls of Trump support in fact flipped over to a near three percentage point lead in live (phone) interview polls, as seen below (note: the use of moderate smoothing might influence what you see here, but the point still stands of a closed gap between the two types of surveys with respect to Trump support). Figures 1 and 2 contain polling results for the two types of surveys for the month of December.

Figure 1: Non-live interview polls for online, automated phone, and IVR/online for the month of December (HuffPost Pollster).
Figure 2: Live interview polls for live phone for the month of December (HuffPost Pollster).

If you extend the start date range to the beginning of the year, a shifted disparity–with live interview polls granting the bigger advantage for Trump–still appear (understandably, though, as the poll tracker still gives greater weight to more recent polls–the influential ones in December).

Consequently, following the decline in the polling divide’s salience, such an explanation for Trump’s success and possibly overstated strength has received less attention. However, it remains a noteworthy aspect of this election, as it holds great bearing over the future course of public opinion research. Moreover, this interview type split still contains importance based on early state elections. Additionally, it has reappeared at the national level: Trump currently gets 39.3 percent of the vote in online/automated polls, but 35.4 percent in live phone interviews.

But back to the early states–and where this divergent polling measurements have become most pronounced.

As of January 9th in Iowa caucus polls, the HuffPost Pollster tracker tabbed Ted Cruz at 36.7% of the vote, Trump at 27.9%, Rubio at 13.5%, and all other candidates with five percentage points or less. Yet even as Cruz begins to consolidate support in the Hawkeye State, there exists a significant difference between how he fares in live interview and online/automated polling.

Figure 3: Live interview polls for live phone (HuffPost Pollster).
Figure 4: Non-live interview polls for online, automated phone, and IVR/online (HuffPost Pollster).

As seen in Figures 3 and 4, the fortunes don’t just change for one candidate between different types of polls. Cruz does nearly eight percentage points better in live interviews–and notably, in a Selzer & Company poll that has one of the best track records in the Iowan caucus environment–whereas Trump does about six percentage points worse.

Interestingly, a similar comparison in the next state that holds an election–New Hampshire–does not return the same conclusive divide. Trump only does incrementally better in non-live interview polls than in live interview ones, though that gap does slightly expand when evaluating solely online polls for the non-live interview category (note: a dearth of data points here should give you pause; most NH polls are live phone interview ones). Perhaps this indicates an overstatement of the effect of different polling methodology.

Yet as we move to the next key election–the primary in South Carolina–a sizable and familiar gap reemerges. Trump does about 10 percentage points worse in live interview polls than he does in non-live ones, while Cruz and Marco Rubio see 3.1 and 2.5 upticks in live interview polls, respectively.

(The Nevada caucus occurs on the same day as the SC primary does, but too few polls–and few recent ones accounting for the different type of interviews–have been conducted of the race, and thus I have excluded it from the analysis.)

The key takeaway from this poll examination shows that while national polls have vacillated in their ability to show the live/non-live interview divide (or understated it at times), the most important race–the Iowa Caucus, because of its imminence–clearly delineates the gap. It could easily prove essential to gauging the GOP primary if what we previously know about different polling methodologies–that live interviews are more predictive and reliable than other forms–holds.


It’s also important to place this finding in the context of divergent meanings of national and state polls.

For one, a national primary never takes place. And while if all state primaries occurred simultaneously this measurement of the national pulse could be valuable, the sequential order of the primaries means that they are hardly independent processes. Rather, each affects the subsequent state primary or set date for several primaries, with media-driven momentum and more importantly, candidate winnowing processes, carrying great weight.

Furthermore, there are a couple more points to consider:

  • negative correlation between state and national polls has been shown to exist, showing why leading in certain polls has advantages over doing so in other ones.
  • Of nearly equal importance, voters sit in different stages of their decision-making process depending on whether the population is the entire country or the early states. Those in Iowa and New Hampshire are further along in their information-gathering process and attention given towards election. Exit polls from prior elections also show how many of these voters actually make their electoral decisions closer to the primary/caucus date than further out. In other words, such a dynamic does not lend support towards seeing candidates with polling leads more than one month out in a position of too much strength. Accordingly, relative to his national standing, these states represent where Trump does poorly: trailing in Iowa and possessing a smaller leading in the Granite State.

In sum, the greater a focus on the states that vote or caucus early, the likely better picture you get of how the GOP primary is actually evolving. And along with the split between live and non-live interview polling, you get a picture of how the strength of Trump in this primary election season remains as overstated as ever.

Revisiting The Live Interview VS. Online/Automated Poll Split And What It Means For The 2016 GOP Primary

3 thoughts on “Revisiting The Live Interview VS. Online/Automated Poll Split And What It Means For The 2016 GOP Primary

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s