The Effects of Different Survey Methodologies and Houses on Donald Trump Approval Rating

This analysis was done in collaboration with G. Elliott Morris, a fellow junior undergraduate student interested in political science and statistics. You can find his blog here and his Twitter page here.

---

While the polling-rich election season may have ended months ago, there remains plenty of debate surrounding public opinion data. Nowadays, the focus revolves around Donald Trump’s approval rating numbers. The data, which has come in the form of 95 different polls containing approval ratings as of February 26th, has been interpreted very differently by different parts of the public–including by the president himself:

The same people who did the phony election polls, and were so wrong, are now doing approval rating polls. They are rigged just like before.

— Donald J. Trump (@realDonaldTrump) January 17, 2017

It’s unlikely data (of any kind) could sway Trump. But the wide-ranging reaction by the public is more understandable as these polls are telling somewhat different stories. Of the 95 approval rating polls that have been conducted, the average net rating is -3.4 points (when you subtract Approve % from Disapprove %, where negative values indicate more people disapprove of Trump than approve of him). But this net approval rating has ranged from as low as -18 points to +18 points. Given how a poll can differ by several factors such as the medium through which it’s conducted and the portion of the public it surveys, this variability should not come as a total surprise. What’s important to do in this case is to measure what qualities of polls lead to a more favorable or unfavorable result for Trump in his approval rating. In this way, interpreting the influx of polls out in the public domain becomes easier.

Trying to gauge effects of different factors all at one makes this situation ripe for multivariate regression analysis. Natalie Jackson of HuffPost Pollster took a key first step on this front, finding that the effects of the Rasmussen poll, polls with registered voters, and polls conducted online had positive significant effects in a regression predicting Trump net approval. Below, in work I collaborated with G. Elliott Morris on, we try to expand on this by first running a more recent regression and doing so for both net approval rating and approval percentage, and then calculating house effects for each pollster.

What Affects Trump’s Approval Rating?

In order to take several different survey characteristics into account all at once and estimate their isolated effects while controlling for all other effects, we used multivariate linear regression. In the table below, we ran models that predicted net approval rating (% approving of Trump in a poll minus % disapproving of Trump), appearing in column 1, and approval percentage (% approving of Trump in a poll), appearing in column 2. The data came from the HuffPost Pollster website. We used a few different independent variables for both models:

1. Survey population (i.e. polling universe): We looked at polls surveying either all adults in the United States, only registered voters, and or “likely voters” (modeled on their likelihood to vote in the next election). The “Adult Population” effect serves as the baseline for the estimate for this variable, with the table below showing the effects of a “Registered Voter Population” and “Likely Voter Population” relative to the “Adult Population” effect.
2. Survey mode: This variable takes into account how a survey is conducted: through a live phone interview, a self-administered online questionnaire, or a mix of interactive voice response and online surveys (IVR/Online). We limit our scope to these three survey modes. Like with the previous variable, we have a baseline–“Live Phone” polls–with effects for “IVR/Online” and “Online” polls measured relative to this baseline appearing in the regression table.
3. Days since the inauguration: Calculated as the days between a poll’s end field date and January 20th, 2017.
4. Poll field time: Calculated as the difference in days between the start and end date of a poll’s period in the field.
5. No opinion percentage: Calculated by subtracting the percent approving and disapproving of Trump from 100, leaving us with people who weren’t sure or had no opinion about Trump in a poll.

	Dependent Variable:
	Net Approval	Approve Pct.
	(1)	(2)
Registered Voter Population (Relative to Adults)	5.209***	2.604***
	(1.467)	(0.733)
Likely Voter Population (Relative to Adults)	5.108	2.554
	(4.237)	(2.118)
IVR/Online Mode (Relative to Live Phone)	9.044**	4.522**
	(4.432)	(2.216)
Internet Mode (Relative to Live Phone)	7.777***	3.888***
	(1.156)	(0.578)
Days Since Inauguration	-0.269***	-0.135***
	(0.047)	(0.023)
Poll Field Time	-0.744**	-0.372**
	(0.370)	(0.185)
No Opinion Pct.	-0.117	-0.559***
	(0.133)	(0.066)
Constant	-2.187	48.907***
	(1.867)	(0.933)
Observations	95	95
R2	0.755	0.849
Adjusted R2	0.735	0.837
Residual Std. Error (df = 87)	4.096	2.048
F Statistic (df = 7; 87)	38.277***	69.823***
Note:	*p<0.1; **p<0.05; ***p<0.01

Both models come up with several statistically significant independent variables, and they explain a large amount of the variation in net approval (74%) and approve percentage (84%) from polls. In model 1 predicting net approval, polls that survey registered voters result in a net 5.2 points more for Trump’s approval ratings than polls that survey all adults. This confirms that when you narrow the population from which you’re sampling from the entire public to only those registered to vote, you’ll end up with respondents more favorable to Trump. Even bigger effects appear for the mode variable. Relative to live phone surveys, IVR/online polls are a net 9.0 points and internet polls a net 7.8 points more favorable to Trump. This makes the early mode effect in Trump approval rating polls very clear: surveys conducted online and without a live interviewer result in much better net approval ratings for Trump than surveys conducted over the phone by live interviewers.

The variable for days since the inauguration is also statistically significant, but in the negative direction: with each day we get further away from the inauguration, Trump’s net approval gets 0.27 points worse. This makes sense given that events during his presidency have likely only tarnished his image rather than improved it, with more occurring as his presidency progressed past his inauguration date. The variable for the amount of days a poll was conducted is also significant and negative, which would indicate that as a poll was fielded for a longer period, the worse Trump’s net approval would result. However, it’s hard to see what actual mechanism is causing this and it’s likely that this variable picks up the effect of another variable (e.g. survey quality), so this significant effect is not very meaningful.

The second model regresses approval percentage–rather than net approval–on all the aforementioned predictors. The same significant effects (coefficients) result and are in the same direction as those in model 1: registered voter populations, IVR/online survey modes, internet only survey modes, fewer days since the inauguration, and shorter field periods result in higher percentages approving of Trump. The variable for no opinion percentage comes up as significant and negative, but this is an artifact of it being related to the dependent variable in this model; approve % and 100 – (approve % + disapprove %) are part of the same 100% of all respondents, so a change in one of these variables will always be negatively associated with a change in the other.

Survey House Effects

Evidence of these mode, population, and period effects are not new. Where we add a new layer of understanding is in calculating survey house effects below.

At this early stage in Trump’s presidency, there aren’t as many approval rating polls to evaluate as we would like. There are currently 34 from Gallup and 23 from Rasmussen, but no other pollster has conducted more than five polls asking about Trump approval. This presents a problem at this early stage, as any house effects we calculate are based on a small sample of polls from a given pollster. Survey house effects are likely fairly variable in these first few months of the Trump era, and effects that appear at this point could easily change over the course of the next few months. Thus, it’s important to keep this caveat in mind when viewing the below house effect calculations–they give only a good early picture at house effects, and not as clear a signal as would get in a few months. That being said, here’s how we carried out this process.

First, we downloaded approval rating data for Trump from the HuffPost Pollster website. Including only data for a poll’s entire population (and not just Republicans or Democrats, for example), we created 17 different variables for the 17 different pollsters who have asked about the president’s approval rating. These variables individually went into different regressions predicting Trump approval percentage (or his net approval rating), along with population (adults–the baseline–registered voters, and likely voters), mode (live phone–the baseline–Internet, and IVR/Online), days since the inauguration, poll field time, and the no opinion percentage (as described before). In this way, for each pollster, we were able to make all other polls the baseline in a regression, and then calculate the effect of each pollster on Trump approval (or net rating) relative to a baseline of all other polls. We term this effect–the coefficient from each different regression for each different pollster–the “house effect” of a given pollster.

The graph below plots the survey house effect for 17 different pollsters when using net rating as the dependent variable in 17 different regressions, from greatest effect against Trump in blue to greatest effect in favor of his net rating in red:

After controlling for various different survey characteristics, PPP polls have the strongest in-house effect against Trump out of all polls measuring approval rating of the new president. On the other end of the spectrum, Rasmussen polls have the strongest in-house effect in favor of Trump in terms of producing greater net approval ratings.

Using approval percentage as the dependent variable in this process doesn’t change much–only the range in coefficient values–as it tells the same story as the above graph:

The below table lays out all the survey house effects (i.e. regression coefficients) for each of the 17 pollsters and for net approval and approval percentage. Let’s use net approval as an example for how to interpret these numbers. Rasmussen polls have an in-house bias that makes Trump net approval 13.8 points better relative to all other polls. Meanwhile, PPP has the opposite effect, as relative to all other pollsters, its in-house bias is 13.8 net points worse for Trump. Gallup, at a net -0.3 points, is currently the poll with the smallest in-house survey effect in either direction. The effects for all the other pollsters follow the same scheme–negative values indicate a survey house bias against Trump, and positive values indicate a survey house bias in favor of Trump.

As mentioned before, a lot of these calculations are tentative. Outside of Gallup and Rasmussen, pollsters don’t have large enough samples of approval ratings for us to assertively conclude house effects. This should just serve as a guide for what to look out for, and which polls have shown early signs of in-house biases. At the moment, Gallup, which has the smallest house effect, is the clearest indicator for Trump approval rating, so it might be worth taking more stock into polls it releases.

Advertisement

The Presence of Survey Mode Effects in Trump Job Approval Numbers

In my last post, I concluded that there was no evidence of a survey mode effect for Donald Trump’s (and for the most part Hillary Clinton’s) polling numbers during the 2016 election. The finding was important as it ruled out the potential for the most simplest way social desirability bias (acting against Trump) could come about–if voters underreported their support for Trump to pollsters due to a hesitation to expressing a socially undesirable opinion to a live interview in phone polls (as opposed to no live person conducting an online poll).

Support levels for Trump, of course, continued to be measured. This time they come in the form of job approval numbers for evaluating the course of his presidency, begging the question whether any survey mode effects appear here as well. Below, I plot Trump’s net approval rating since the inauguration broken up by survey mode–live phone polls versus online/IVR polls. I include a label for the Rasmussen poll, which has proven far more favorable to Trump during the campaign, been more favorable to him for gauging his job approval, and historically has an in-house bias towards Republicans.

Here it becomes obvious that there does exist a consistent and fairly sizable mode effect. As would be expected if there was in fact a social desirability bias effect in play, Trump sees a much better net approval rating in Internet/IVR polls than in those conducted through live phone interviews. The survey mode gap began at around a seven-point gap, approached 15 points within a week or so (i.e. live phone polls and online ones differed by as much as 10-15 points at one time), and now stands at around eight points. Clearly, it’s impossible to conclude a social desirability effect from something like this, but it certainly does leave room for this possibility. In some part, however, the Rasmussen poll–conducted through interactive voice response (IVR)–might be driving this difference more than mode itself. It consistently finds a better approval rating for Trump than almost all other polls. When it’s removed, the survey mode gap shrinks, but does not go away entirely, as can be seen in the below graph that does not include Rasmussen poll data for generating the smoothed approval rating trend lines.

The survey mode effect still bounces around a bit but is noticeably smaller. The latest polls suggest there is very little gap in what different survey modes estimate Trump’s approval rating to be.

Finally, to check whether any differences exist by the percent saying they approve or percent saying they disapprove of Trump’s job performance–instead of looking at just the net rating that subtracts these percentages–the below plot distinguishes between these two options. This returns to including polling data from Rasmussen as well.

Survey mode effects remain intact regardless of whether you look at trends in only approval percentage or only disapproval percentage. Relative to live phone polls, people are more likely to say the approve of Trump and less likely to say they disapprove of Trump in online polls. The gap seems a little larger for the “approve” percentage, but not by much. The point stands that survey mode effects are present in Trump job approval numbers with online polls giving him better ratings, a result driven a good amount but not completely by one poll (Rasmussen).

---

Finally, one other aside: the above graph also makes clear that recent changes and decline in Trump’s approval ratings have more to do with more people disapproving of him than fewer people approving of him. The percent that approves of him, as you can see in the left-hand side of the graph, stays more or less constant since the inauguration. The movement instead occurs on the right-hand side, as more people say they disapprove of Trump (regardless of mode) over this time period. In other words, it may be that Trump is not losing people who already approved of him, but rather that he has attracted more disapproval from people who previously did not give a rating. It should be noted that there are always about 5-10 percent who don’t give an approval assessment in these polls, so there is this pool of Americans–likely in the partisan and ideological middle ground–that could sway things like this. This point contrasts from the one I concluded in evaluating Trump’s falling favorability ratings about a month ago, where that dynamic had more to do with people leaving the “favorable” pile than greater amounts saying they had an “unfavorable” opinion of Trump.

Survey Mode Effects in 2016 Election Polling and the Lack of Bias Against Trump

Survey mode effects–the idea that results from surveys differ based on whether the survey is conducted by a live person interviewer through a phone call or self-administered online–has been an oft-discussed topic during the 2016 election season. It first came up during the GOP primary and again gained attention during the general election season. Perhaps most importantly, it’s been frequently used to examine the potential for social desirability bias effects in estimating candidate support before the election. Comparing live phone polls–where this bias would materialize as a result of talking with another human and choosing to not reveal socially undesirable opinions–and online polls–an anonymous setting that does not create the same level of bias effects–could reveal the presence (if any) of this bias. This question of whether social desirability bias played a role in polling error and more specifically for underestimating Trump support is a lot more complicated than a simple mode comparison (see section 4 in my article here for a quick review of competing arguments). Nevertheless, it’s worth looking into mode effects such as these as a clear-cut way of getting at social desirability bias–and whether there was a “shy Trump” effect.

Patrick Egan did a good initial job of this at the Monkey Cage in tracking Hillary Clinton’s percentage support margin lead over Donald Trump from July until the election. He finds no major difference between live phone polls and online ones, and rightly concludes that no evidence of social desirability bias acting against Trump exits. One thing I wanted to check, however, is how these mode effects developed not just in terms of candidate support margin over the course of the campaign (i.e. Clinton % minus Trump %), but also in terms of each candidate’s individual support level. Were differences in mode greater in estimating one candidate’s share of support than another’s? That’s what I show in the below graph.

Donald Trump Approval Rating: 2/6/2017 Update

Just a quick analysis here, but I took a look at the 36 approval ratings for Donald Trump that have come out since his inauguration. 12 of these had partisanship breakdowns, and so I wanted to see how not only Trump’s approval ratings broke down by party, but also if they differed by population: results for all adults as opposed to results only among registered voters. In the past, I’ve found Trump’s favorability ratings to differ a bit by population, so it’s always worthwhile to consider the universe from which a pollster is sampling. Here I also want to break it down by party and population, making for six subgroups in all, in order to check where Trump’s recent decline in approval–since around the end of his first week as president–is coming from.

In the above graph, the red line signifies the percentage in each subgroup that approves of Donald Trump’s job performance, and is represented as a positive value. The blue line signifies the percentage saying they disapprove and appears as a negative value. The black line essentially adds these percentages together to plot the net job approval rating for Trump. Each large hollow circle marks a poll graphed by its end field date on the x-axis.

Among registered voters, the approval ratings given by the two partisan groups don’t change much over this time period. Among Independents, Trump’s approval rating declines slightly around the 24th and 25 of January, but that doesn’t have much to do with his recent collapse. Rather, approval ratings among the general population–when surveying all American adults–have captured this decline. Notably, not much changes among Republican identifiers, as even recent controversies have not deterred these partisans from supporting their party leader and new president.

Instead, the decline in Trump’s approval rating seems to be driven most by worsening evaluations among Independents, as well as among Democrats but to a lesser degree in this case. Two recent polls in the above graph have played the biggest role in ushering in this decline: a CBS News one and a CNN/ORC one. Moreover, while it’s hard to parse these relatively small differences with few data points, it’s worth noting the recent decline appears to be in large part due to more adults saying they “disapprove” (blue line) of Trump rather than fewer adults saying they “approve” (the red line). In other words, it looks like more people have moved into the “disapprove” column after being in the “don’t know/no opinion” column than people moving out of the “approve” column. All of this, of course, will become clearer once more data gets collected.

---

Two things to keep in mind when looking at the above graph:

• The partisanship data was gathered from the HuffPost Pollster website. Pollsters vary in their definitions of party members, specifically on the question of whether to group Independents who lean toward a party with that party rather than as Independents–some group these leaners, others don’t. I find it inexplicable to not group Independent leaners when evaluating survey question breakdowns by party ID, given the swath of political science research supporting the decision to group leaners and my own look at this data doing so too. (Quick summary: Independent leaners behave almost identically as regular partisans–why not classify them as such?) Regardless, the Pollster data doesn’t specify this grouping decision, so I’ll just have to live with the form in which the data comes here–a slightly less meaningful form, but to no fault of the Pollster website.
• When I break up results by party ID and population, that leaves between five and seven data points (polls) to cover the span of a few weeks. The loess smoother is thus obviously not very meaningful here given the small amount of polls; it’s included to give just a rough sense of polling movement.

Clarifying Polling Error in 2016 (Dartmouth Political Times)

I recently came across an article at one of my school’s publications about polling error in the 2016 election. The piece had several misconceptions about polls, forecasts, and error that are likely still shared by many beyond just the article’s author, so I wrote a response to it that you find here at the Dartmouth Political Times.

Pro-Life Supporters and Resistance to the Trump Presidency

One week after the inauguration of Donald Trump as president and days of tumult and resistance to the new presidency, the March for Life was held in Washington, D.C. Around the time of this pro-life event and the Women’s March on Washington right before, many debated whether common ground between these advocates and more liberal ones–such as pro-choice supporters–was possible as an alliance of opposition to Trump.