Partisan Nonresponse and Obama Approval Polls

It seems that once you start looking for differential partisan nonresponse, it turns up in a lot of places. That’s the case not only for current and relatively recent data, but also–to a lesser degree–for data from the past decade, as I’ll cover in this post with President Barack Obama’s approval rating polls.

In my last post, I showed a fairly strong positive relationship has formed between the partisan makeup of a poll and Trump job assessment in approval ratings polls over the last few months. Most interestingly and indicative of partisan nonresponse bias effects, the relationship does appear for polls that make no effort to adjust for the partisan character of their sample in weighting methodology, but largely does not appear for polls that weight by party identification or past vote. The pattern among polls that don’t include this type of weight suggests that the partisan composition of a poll–rather than solely true opinion movement–factors substantially into determining Trump approval.

Hypothetical alternative explanations exist for this result. Polls vary in their sampling approaches, weighting schemes, and even how they report data, such as partisan distribution. These attributes in one way or another could possibly influence the relationship I observed. While I can’t account for everything, it’s worth noting that the statistically significant positive effect of partisan makeup on net Trump approval does hold up in a multivariate model controlling for other poll characteristics. Another explanation could entail an idea that individual party identification moves in tandem with presidential approval. Of course, this stands in stark contrast to the wealth of research demonstrating the stability of partisanship. Some research has shown there’s more individual level partisanship movement than widely assumed, but that’s not nearly enough to make this alternative plausible. (More importantly, for this to matter, there must be drastic real shifts in aggregate partisan distribution–not just individual level shifts that can cancel out–which clearly do not occur.)

Bivariate relationships–between partisan composition and public opinion that heavily involves party politics–can thus be very telling. Just as I did with Trump approval rating polls earlier, I wanted to check the party composition vs. net approval relationship, this time using approval rating polls of Obama during his presidency. Once again, I turned to HuffPost Pollster’s extensive database of polls that span the start of Obama’s presidency in 2009 to its end in 2017. From here I was able to record my independent variable (the difference in unweighted percentages of Democrats and Republicans) and my dependent variable (the difference in percentages approving and disapproving of Obama as president). For more detail on methods, and an important note about data collection (in paragraph 7), check my last post that covers very similar ground.

In the below graph, I plot this relationship. Polls that include some weighting adjustment for ideology or past vote choice appear on the left-hand side in red. There were very few non-YouGov pollsters that fell in this category, so for simplicity and the purposes of maintaining large enough samples of polls from individual survey houses, I only include YouGov for this classification. Polls that don’t include these type of weights–i.e., that don’t adjust for nonrandom partisan selection into polls–fall on the right-hand side of the graph in black.

Little relationship between partisan composition (a proxy for partisan nonresponse patterns) and net Obama approval exists among YouGov polls. Apart from a few polls that contain net even and pro-Republican makeups (that play a big role in turning the line of best fit slightly negative), the unweighted partisan distribution of YouGov polls stays fairly stable. On the other hand, among polls that don’t weight by ideology/past vote, a weak to moderate positive relationship (correlation coefficient of 0.29) forms between partisan makeup and Obama approval. The R-squared is small (0.08), though, especially compared to the same number but for Trump approval polls I found before (0.45).

Nevertheless, while not as strong, there does appear to be at least some effect of a poll’s partisan makeup on its ultimate outcome of interest, evaluation of a president. As noted before, this pattern suggests evidence of differential partisan nonresponse bias, wherein partisans non-randomly select into/out of polls in a way that makes results contingent on how many partisans on each side choose to take a poll. Notably, this differs to some degree from polls that try to account for this differential nonresponse activity (i.e. YouGov), which show less of a relationship. While not nearly as compelling as with Trump approval polls, it does seem that even during the Obama presidency with approval ratings, polls that don’t account for varying partisan willingness to take surveys suffer from bias.

Advertisement