Possible Evidence of Differential Partisan Nonresponse in Trump Approval Polls

Differential partisan nonresponse–the idea that survey response rates vary by partisanship–has received a lot of attention as of late (I had some thoughts on this subject a while ago too). Most notably, strong evidence exists that this specific pattern can explain much of the opinion movement seen in pre-election polling during the campaign period, and might be responsible for polling error in this past election. Nonresponse bias of any kind has been considered as a source for election polling error as well.

The differential willingness to partake in surveys among Democrats and Republicans in response to current events is more intense during an election season, but likely not confined to it. Politicization of public opinion polls (to some extent) and many other factors such as trends in social trust could mean that partisan nonresponse bias is becoming more widely entrenched. The current political mood of a country and favorable events for one’s political group may regulate survey response willingness for salient topics beyond vote intention, such as presidential approval of Donald Trump.

This phenomenon has important implications outside of just consideration for survey research practices. If the amount of Democrats or Republicans selecting into and out of polls shapes outcomes of interest, movement in public opinion become more illusory and an artifact of survey error. Corrections for this error, such as weighting to benchmarks for party identification or past vote distributions, seem promising but have remained far from embraced by the survey research community (in part, it depends on survey data types, such as panel or cross-sectional surveys).

In short, the questions over the severity of partisan nonresponse bias and how to correct for it are incredibly important but largely unsettled. The best current data to help speak to this issue lies in Trump approval rating polls. The Trump presidency has thus far been plagued by controversies scattered across his roughly half-year in office, introducing potential for partisan nonresponse. For example, at various points in the last few months, it’s very conceivable that Republicans heard bad news about their party’s president and became less willing to participate in surveys–likely about a topic unfavorable to them and their party–that were conducted around the time of this bad news.

To test this general idea, I compared the partisan composition of a poll and the level of Trump support in the same poll. I took data from HuffPost Pollster‘s database, which included all Trump approval polls conducted since his inauguration as president, and narrowed down to only look at polls that reported partisan breakdowns of Trump approval and the total number of partisans contained in a poll. Not all polls report partisan breakdowns and thus the Pollster data does not contain this data for several polls. Polls that report partisan breakdowns tend to be conducted more online than through live phone interviews. Besides this, it’s unlikely the polls with breakdowns I include and the polls without breakdowns I exclude systematically differ in many other ways and to the point that it would introduce serious bias (in a context where I want my results to speak to a phenomenon occurring in all polls).

For each poll–124 in total that included partisan subgroup information–I then calculated net Trump approval (the percentage approving of him minus the percentage disapproving of him in the aggregate) as well as the unweighted percentage of Democrats and Republicans in each poll. Using these latter pieces of data allowed me to create a partisan distribution metric: the percentage of a poll that was net Republican (Republican percentage of a poll minus Democratic percentage of a poll).

One note on this last variable: every pollster outside of Ipsos/Reuters in my data broke down Trump approval using a party identification measure that did not group Independents who leaned toward a party with that party. In most contexts, this offers a very deceptive picture of the partisanship of survey respondents (I explain why here). However, the “Republican % Net” variable I created should suffice as an adequate proxy for the partisan composition of a poll (the partisan distribution difference likely changes little on account of grouping or not grouping leaners).

In the below graph, I plot this partisan composition variable (net Republican percentage) against net Trump approval percentage. If Trump approval increases (decreases) as the number of Republicans in a poll increases (decreases), then this would roughly confirm a story of differential partisan nonresponse. I also add another wrinkle to this process in measuring the same relationship by breaking up the plot by whether a poll was done by a pollster that weights by party identification benchmarks (ICITIZEN and IBD/TIPP) and 2016 vote choice (YouGov, in partnership with either The Economist or HuffPost), or not. Adding this type of weight works to correct for differential partisan nonresponse, so comparing polls that do not include any correction to those who do provides the clearest picture.

In accordance with the presence of differential partisan nonresponse, the amount an unweighted poll sample is Republican is closely linked to Trump’s net approval rating in polls that do not weight by party or past vote. Specifically, the partisan difference variable explains 45 percent of the variation in Trump approval. As more Republicans select into taking a poll, the better the Trump approval rating becomes. Importantly, this would suggest that the key outcome of interest–how many Americans approve of Trump as president–hinges considerably on how many Republicans and Democrats choose to take a poll. This would break with the implicit assumption most people hold when evaluating polls that volatility in Trump approval ratings reflects actual change in people’s opinion of him. When Trump’s approval declines, for example, this might not indicate fewer Americans approving of him but rather fewer Republicans taking surveys–and the pollsters themselves doing nothing to correct for this bias.

But for the pollsters who do attempt to address partisan nonresponse bias–on the left side of the above graph–the same pattern does not hold. Because these pollsters use different methods to weight by party/past vote, I show the plotted relationship between partisan composition and Trump approval separately for the three survey houses (the two YouGov polls in red, IBD/TIPP in blue, and ICITIZEN in green). For all three of these pollsters that use unorthodox weighting methods, an equally strong relationship between partisan distribution of a poll and Trump net approval does not exist, and importantly not in the positive direction as seen in the right-hand side of the graph.

Adding weights for party ID or 2016 vote choice also seems to limit the variation in partisan composition from one poll to the next. Outside of one outlier, polls from YouGov all show a similar partisan distribution. For IBD/TIPP that’s less so the case, but for ICITIZEN the lesser variability is very clear. This variability aspect as well as comparing the plotted relationships before shed light on an important phenomenon: while pollsters that do not weight by party or past vote see a strong relationship between partisan distribution and Trump approval in their polls–indicative of differential partisan nonresponse effects–pollsters that do employ this weighting method seem to better overcome nonresponse bias problems (as expressed by the weak relationships in the left half of the graph).

It should be noted that all of these results materialize at a very crude, cross-sectional level. Panel data and studies, for example, would offer the most definitive account of partisan nonresponse trends. Nevertheless, it should remain notable that a strong relationship indicative of nonresponse bias still forms using this limited data. To round out this analysis, I also model Trump approval and account for other poll characteristics below using linear regression for only polls that do not weight by party/past vote. In the first column, I show the output for the bivariate regression illustrated in the plot–regressing Trump net approval percentage on Republican net percentage advantage in each poll. In models 2-4 I add in controls, such as an indicator for polls that use a registered voter universe relative to a baseline of all adults, and indicator for polls that use live phone surveys with online surveys as the base, and the number of respondents in a poll.

The first model shows that as the net Republican percentage in a poll moves one point up, Trump’s net approval percentage increases by 1.4 points. This positive statistically significant relationship is the same one expressed in the plot above. Adding in the various controls one by one in subsequent models decreases the effect size of the partisan composition variable on Trump approval a bit, but in each specification, the relationship remains positive and statistically significant.

There are of course other factors that I did not account for and others I could not measure, but the regression analysis still serves to bolster evidence behind the presence of this relationship–suggestive of partisan nonresponse bias effects–in surveys conducted by pollsters who don’t weight by party or past vote. These results of course should not imply that weighting by partisanship or past vote is the definite answer to solving the problem of differential partisan nonresponse. However, given the goal of polling remains to accurately capture opinion and its shifts, and in light of this considerable impact of a poll’s partisan makeup on a poll’s results, differential partisan nonresponse should be treated more seriously and new weighting methods considered as one option to address the issue.

---

7/19/17 edit: Some of this analysis was featured in a post on the Washington Post’s Monkey Cage blog, which you can find here.