More on Democratic Leftward Turn Survey Experiment

I appreciate all the discussion and critique surrounding my New York Times article that came out earlier this week. It has made me think harder about the structure of my study design and implications of my findings, and I now realize where I could have been clearer in presentation of results (though of course, there are limits to how much detail and rationale I can include in a general audience piece). I wanted to briefly clarify some things in this blog post, and specifically address three common critiques that arose:

1. Representativeness of progressive policies included in the treatment (e.g. here).
2. Lack of Republican (or counter-) messaging (e.g. here).
3. Reactions to leftward shift prime/frame vs. policies themselves (e.g. here).

There’s all kinds of flexibility and motivations one can take to designing a survey experiment like the one I ran. The main impetus for my study was the leftist rhetoric on display in the first few Democratic debates, as I describe in the article. The primary debates were not the first or only instances that signaled growing progressive rhetoric (for example, see this coverage earlier in the year), but they notably were on a much more national stage than before. Given this potential for people to become more aware of such liberalizing rhetoric, and this context in which the Democratic Party was publicly (re)defining itself, the debates–and subsequent coverage–are arguably emblematic of the leftward turn the party has been taking.

Even if this doesn’t sound reasonable, I still want to make clear that the springboard for my study was the primary debates, and how the growing progressivism is conveyed in this context. After establishing that I wanted to capture reactions to the debates and leftist rhetoric therein, I then had to decide how to present information about this. One option that might immediately come to mind is to simply list excerpts of what candidates said in the debates. But I would argue this does not attend to a key tenet of survey experimental design: realism. I want to mirror the way in which many voters might be encountering this information in the real world. Voters–especially those not identifying as Democrats–are likely not watching the debates, making a pure information approach (e.g. using candidate excerpts) unrepresentative. Instead, voters likely hear about the debates and receive information about them via news media outlets. As I say in the original article, survey experimental “content was drawn directly from real news coverage.”

Again emphasizing how voters might encounter debate-related information in the real world, I shape my experimental treatment to mirror real debate coverage from major, popular online news outlets. People are more likely to encounter news from these outlets, and as mostly neutral sources, this still retains important “pure information” treatment qualities (i.e. I do not want to turn this into a framing experiment). (Also note that the degree of partisan selective exposure in news consumption appears to be overstated, making my choices of more neutral, mainstream outlets justifiable.)

I kept all of this in mind when deciding what news coverage to base my experimental design on. I’ve organized various specific post-debate news articles that I drew on in creating my treatment condition; they can be found at this link (including the survey treatment text for comparison). Headlines from this post-debate coverage include things like “Democrats Veer Left,” “What the Democrats’ Turn Leftward Means for the Party’s Chances,” “Debate shows how leftward [the Democratic Party] has moved,” “Liberal Democrats Ruled the Debates,” and “Democrats lead with their left.” All the post-debate articles devoted a heavy focus to proposals on decriminalizing unauthorized border crossings and expanding undocumented immigrants’ access to government services.

I believe the aforementioned critiques are addressed in this greater discussion of rationale behind my design and specific news articles I drew on. In other words, I argue that my design accurately reflects how many voters might have encountered information about and coverage of the Democratic debates. Here’s how this effort to reflect real world news coverages addresses the three critiques above:

1.

A few points from one of the representativeness critiques:

• “..policies can be unpopular without being salient.”
• “There is no value, however, in casting these three policies as definitive of the Democratic Party’s ideological shift.”
• “By equating the political viability of an arbitrary (and unusually unpopular) batch of left-wing policies with that of Democrats “moving left” in general..”

I neither arbitrarily chose the policies included in the experimental treatment, nor did I intentionally choose the least popular left-wing policies. As the above cited articles–and the content of the debates themselves–make clear, these policies were clearly the most salient during the debate and amid post-debate news coverage. I also made no claim that these were representative of the party’s broader leftward shift. What I included in my design was a distilled version of a lot of these post-debate news pieces. Of course, whether these types of policies are repeatedly made salient on a national stage over the rest of the campaign will likely determine whether Democrats will suffer an electoral cost in the end.

2.

Clarifying my aim to measure reactions to the Democratic debates and debate coverage helps explain why I focused on one-sided messaging in this study–in the information environment following the Democratic debates, a Republican (or Trump) message is not relevant or salient (not least because the example news articles I collected hardly mention that alternative right-wing message, if at all). Given the purpose of this study, I did not find it relevant to include this alternative message about right-wing policies and rhetoric from Republicans and Trump. Many people felt this aspect was lacking in my design, but such thinking came with a general election context in mind. Needless to say, we are currently not in the general election, where the information environment would otherwise feature two prominent campaign messages; Democrats are arguably the party more so in “campaign mode” at the moment. Lastly, the leftist turn as expressed in the debates is what’s really new in this environment (Republicans/Trump have been espousing right-wing/extreme rhetoric for years now), and I would argue it’s justifiable to devote more attention to this new phenomenon.

3.

I think the distinction between the leftward turn prime–saying the debates “have illustrated how far to the left the party has moved”–and describing specific policies that candidates supported is really important and opens up an avenue for future research. To be clear, I do not mean to equate these two portions of the treatment. Once again, I aim to emulate how my sample of real world news coverage presented the debates. As mentioned above, this leftward turn prime and inclusion of specific candidate policies always went hand-in-hand in the news articles I drew on. I used what might be called a “bundled treatment”–combining the prime and policies together and thus making their contributions to the observed effects inseparable in this context–but only because this information appeared as a bundled treatment in the real world as well.

This nevertheless leaves an important question unanswered: is the negative reaction by Independent voters driven more by the leftward turn prime, or by the policies themselves? The prime might be more important if Independents are symbolically opposed to ideological or partisan extremity, but the policies could be at play given their focus on immigration-related matters and how these considerations have sparked electoral backlash in the past. As I said, there are clear paths for future research in adjudicating between these potential underlying mechanisms. For example, one option is running an experiment with the same outcomes but new conditions–a leftward turn prime alone, specific immigration leftist policies, and specific non-immigration leftist policies (plus a control).

---

Hopefully that better clears up the thinking that went into the experimental design. Feel free to contact me about anything else.

I’ll end with one last minor point. As a few critics explicitly said and others may have inferred, one might see the results of this study and believe that I started with a certain viewpoint or narrative and from there sought out data to confirm it. Beyond the fact that this is not how I do research (or was trained to do it in political science), I want to emphasize that I was completely open to all possible outcomes to this study–leftist rhetoric engendering negative reactions, positive ones, or none at all. There of course was widespread speculation that would suggest a negative reaction, but speculation like this is often flawed, and there were reasons to discredit it specifically in this case (e.g. the strong current in political science saying candidate policies do not factor into voter decision-making). To this end, I prepared for the other likely result to this experiment–null effects of exposure to leftist rhetoric. Thinking ahead, in order to make the case for a convincing null effects story, I would need to prove that individuals properly received the treatment–that they read the news snippet I gave them at least with some care and recognized the Democratic Party as more liberal, for example–but did not act on it in their subsequent judgments. Indeed, I used a manipulation check as noted in another blog post (and had a factual recall question that was mistakenly not included by the person running my experiment) with all of this in mind, and thus was ready to report either of these potential results.

Advertisement