The Presence of Survey Mode Effects in Trump Job Approval Numbers

In my last post, I concluded that there was no evidence of a survey mode effect for Donald Trump’s (and for the most part Hillary Clinton’s) polling numbers during the 2016 election. The finding was important as it ruled out the potential for the most simplest way social desirability bias (acting against Trump) could come about–if voters underreported their support for Trump to pollsters due to a hesitation to expressing a socially undesirable opinion to a live interview in phone polls (as opposed to no live person conducting an online poll).

Support levels for Trump, of course, continued to be measured. This time they come in the form of job approval numbers for evaluating the course of his presidency, begging the question whether any survey mode effects appear here as well. Below, I plot Trump’s net approval rating since the inauguration broken up by survey mode–live phone polls versus online/IVR polls. I include a label for the Rasmussen poll, which has proven far more favorable to Trump during the campaign, been more favorable to him for gauging his job approval, and historically has an in-house bias towards Republicans.

Here it becomes obvious that there does exist a consistent and fairly sizable mode effect. As would be expected if there was in fact a social desirability bias effect in play, Trump sees a much better net approval rating in Internet/IVR polls than in those conducted through live phone interviews. The survey mode gap began at around a seven-point gap, approached 15 points within a week or so (i.e. live phone polls and online ones differed by as much as 10-15 points at one time), and now stands at around eight points. Clearly, it’s impossible to conclude a social desirability effect from something like this, but it certainly does leave room for this possibility. In some part, however, the Rasmussen poll–conducted through interactive voice response (IVR)–might be driving this difference more than mode itself. It consistently finds a better approval rating for Trump than almost all other polls. When it’s removed, the survey mode gap shrinks, but does not go away entirely, as can be seen in the below graph that does not include Rasmussen poll data for generating the smoothed approval rating trend lines.

The survey mode effect still bounces around a bit but is noticeably smaller. The latest polls suggest there is very little gap in what different survey modes estimate Trump’s approval rating to be.

Finally, to check whether any differences exist by the percent saying they approve or percent saying they disapprove of Trump’s job performance–instead of looking at just the net rating that subtracts these percentages–the below plot distinguishes between these two options. This returns to including polling data from Rasmussen as well.

Survey mode effects remain intact regardless of whether you look at trends in only approval percentage or only disapproval percentage. Relative to live phone polls, people are more likely to say the approve of Trump and less likely to say they disapprove of Trump in online polls. The gap seems a little larger for the “approve” percentage, but not by much. The point stands that survey mode effects are present in Trump job approval numbers with online polls giving him better ratings, a result driven a good amount but not completely by one poll (Rasmussen).

---

Finally, one other aside: the above graph also makes clear that recent changes and decline in Trump’s approval ratings have more to do with more people disapproving of him than fewer people approving of him. The percent that approves of him, as you can see in the left-hand side of the graph, stays more or less constant since the inauguration. The movement instead occurs on the right-hand side, as more people say they disapprove of Trump (regardless of mode) over this time period. In other words, it may be that Trump is not losing people who already approved of him, but rather that he has attracted more disapproval from people who previously did not give a rating. It should be noted that there are always about 5-10 percent who don’t give an approval assessment in these polls, so there is this pool of Americans–likely in the partisan and ideological middle ground–that could sway things like this. This point contrasts from the one I concluded in evaluating Trump’s falling favorability ratings about a month ago, where that dynamic had more to do with people leaving the “favorable” pile than greater amounts saying they had an “unfavorable” opinion of Trump.

Advertisement