James Bisbee
Research
Social Media and Politics
A Descriptive Analysis of how Political Scientists Use Twitter in 2019
with Jennifer M. Larson and Kevin Munger (2022)
Perspectives on Politics
Abstract: Knowledge creation is a social enterprise, especially in political science. Sharing new findings widely and quickly is essential for progress. Scholars can now use Twitter to rapidly disseminate ideas, and many do. What are the implications of this new tool? Who uses it, how do they use it, and what are the implications for exacerbating or ameliorating existing inequalities in terms of research dissemination and attention? We construct a novel dataset of all 1,236 political science professors at PhD-granting institutions in the United States who have a Twitter account to answer these questions. We find that female scholars and those on the tenure track are more likely to use Twitter, especially for the dissemination of research. However, we consistently find that research by men shared on Twitter is more likely to be passed along further by men than research by women.
Supply-Side Polarization: How YouTube's Recommendation Algorithm Pushes Real People Content
with Megan A. Brown, Richard Bonneau, Joshua A. Tucker, and Jonathan Nagler (2021)
KDI Symposium on Polarization in South Korea and the United States
Abstract: To what extent does the YouTube recommendation algorithm push users toward more ideologically congruent content? Existing research on this question has relied on anonymous browsers paired with user-agnostic APIs, and has found very little evidence that the recommendation algorithms are to blame. We surveyed real YouTube users to navigate from a provided starting video and clicked on recommended videos according to a simple rule (i.e., always click the first video, second video, third video, etc.). We demonstrate that the YouTube recommendation algorithm does, in fact, push real users toward more ideologically congruent content by showing that Republican users are exposed to more conservative content than are Democratic users. Furthermore, we show that this difference is entirely due to the algorithm recommending more conservative content to Republicans than what would be shown due to statistical mean reversion. Our analysis provides an ecologically valid estimate of the degree to which recommendation algorithms influence the ideological content that the public consumes from the supply side.
Election Fraud, YouTube, and Public Perception of the Legitimacy of President Biden
with Megan A. Brown, Angela Lai, Richard Bonneau, Joshua A. Tucker, and Jonathan Nagler (2022)
Journal of Online Trust and Safety
Abstract: Skepticism about the outcome of the 2020 presidential election in the United States led to a historic attack on the Capitol on January 6th, 2021 and represents one of the greatest challenges to America's democratic institutions in over a century. Narratives of fraud and conspiracy theories proliferated over the fall of 2020, finding fertile ground across online social networks, although little is know about the extent and drivers of this spread. In this letter, we show that users who were more skeptical of the election's legitimacy were more likely to be recommended content that featured narratives questioning the election's legitimacy. Our findings underscore the tension between an ''effective'' recommendation system that provides users with the content they want, and a dangerous mechanism by which misinformation, disinformation, and conspiracies can find their way to those most likely to believe them.
Estimating the Ideology of Political YouTube Videos
with Megan A. Brown, Angela Lai, Richard Bonneau, Joshua A. Tucker, and Jonathan Nagler (2024)
Political Analysis
Abstract: We present a method for estimating the ideology of political YouTube videos. As online media increasingly influences how people engage with politics, so does the importance of quantifying the ideology of such media for research. The subfield of estimating ideology as a latent variable has often focused on traditional actors such as legislators, while more recent work has used social media data to estimate the ideology of ordinary users, political elites, and media sources. We build on this work by developing a method to estimate the ideologies of YouTube videos, an important subset of media, based on their accompanying text metadata. First, we take Reddit posts linking to YouTube videos and use correspondence analysis to place those videos in an ideological space. We then train a text-based model with those estimated ideologies as training labels, enabling us to estimate the ideologies of videos not posted on Reddit. These predicted ideologies are then validated against human labels. Finally, we demonstrate the utility of this method by applying it to the watch histories of survey respondents with self-identified ideologies to evaluate the prevalence of echo chambers on YouTube. Our approach gives video-level scores based only on supplied text metadata, is scalable, and can be easily adjusted to account for changes in the ideological climate. This method could also be generalized to estimate the ideology of other items referenced or posted on Reddit.
Donald Trump, Partisan Motivated Reasoning, and Covid-19: A Partisan Filtered Theory of Backlash
with Haohan Chen, Richard Bonneau, Joshua A. Tucker, and Jonathan Nagler (2024)
Working Paper
Abstract: The polarization of attitudes on Covid-19 between liberals and conservatives in the United States was, and continues to remain, stark. But to what degree was Donald Trump individually responsible for this phenomenon? We exploit seven high-salience cues about the severity of Covid-19 from former President Trump over the course of 2020 to document a causal effect of Trump's efforts to downplay the seriousness of the pandemic on both liberal and conservative beliefs. We show that the most influential of these cues occurred very early on in the year when former President Trump compared Covid-19 to the seasonal flu. Subsequent cues, including the national State of Emergency, former President Trump's ``liberate'' tweets, the first time he wore a mask in public, and his positive Covid-19 diagnosis, all produced smaller albeit significant effects on public sentiment, which we measure with a bespoke sample of over 600,000 random Twitter users. We argue that the divergence in the public's concern about Covid-19's health risks following the flu comparison were driven largely by liberals growing more concerned about Covid-19, and propose an explanation for this ``backlash'' effect that recognizes elite cues are never experienced in a vacuum. Instead, Trump's comparison was reported on by an ideologically diverse set of media accounts, whose coverage of Trump's cue influenced how their followers updated their concern, a framework we refer to as ``Partisan Filtered Backlash''. Our findings speak to the process by which opinions on Covid-19 became polarized, to the broader debate between the Folk Theory of representation and Partisan Motivated Reasoning, and propose a novel framework for understanding how elite cues are received in the information-dense reality of online social media environments.
Punishing Politicians on the basis of Covid-19
with Haohan Chen, Richard Bonneau, Joshua A. Tucker, and Jonathan Nagler (2024)
Working Paper
Abstract: The political polarization over the Covid-19 pandemic challenges existing theoretical expectations about where and when we should observe evidence of partisan motivated reasoning (PMR). Put simply, voters should adhere to democratic norms of accountability by punishing politicians who deviate from their preferred policies when (1) those policies are highly salient, (2) where the parties have not clearly differentiated themselves, and (3) where the information environment is rich. Covid-19, particularly in the beginning of the pandemic, adhered to all three of these dimensions, yet there is a growing consensus in the literature that it is a political issue. We push back on this conclusion by demonstrating that there is important over-time variation in the strength of democratic accountability which is consistent with the scope conditions of where and when we should expect to observe more or less partisan motivated reasoning. Specifically, we show that voters were more likely to disapprove of their governors' performance on Covid-19 when their governor diverged from the level of concern of their constituents, that these patterns were stronger toward the beginning of the pandemic, that these patterns were exacerbated by spikes in the change in weekly, and that the predictive power of the ``concern gap'' and objective facts declined as the predictive power of partisan congruence between voters and their governors increased. Far from being a uniquely polarizing pandemic, our results suggest that Covid-19 evolved according to the predictions of partisan motivated reasoning.
Identifying the Causal Effect of Twitter's Interventions on the Spread of Misinformation
with Swapneel Mehta, Richard Bonneau, Joshua A. Tucker, and Jonathan Nagler (2024)
Working Paper
Abstract: We estimate the causal effect of interventions deployed by Twitter to limit the misinformation being shared by former President Donald J. Trump via his Twitter account in the months following the United States Presidential election on November 3, 2020 leading up to the Capitol riot on January 6, 2021. We study the multi-platform effects of Twitter's interventions considering the broader impact on audiences extending beyond Trump's following on a single social network. In particular, we employ a trajectory balancing approach for time-series cross-sectional data to estimate the causal effect of warning labels on the retweets and favorites received by a tweet posted on Twitter, and shared on Facebook, Reddit, and Instagram. We also estimate the causal effect of the restriction on sharing of tweets by Twitter on conversations referencing them posted on other platforms. Our findings reveal that warning labels cause a milder Streisand effect than previously estimated on Twitter. On the other hand these interventions seem to bring about a reduction in public conversations about it on other platforms while contributing to an increase in private conversations thereof.
Synthetic versus Human YouTubers: Evaluating the usefulness of an LLM to replicate toxic humans online
with Daniel Rosen (2024)
Working Paper
Abstract: The advent of large language models (LLMs) has raised both threats and opportunities for the study of online discourse and politically relevant information. Yet the contours of these abilities remain understudied. In this paper, we fine-tune the currently cutting-edge publicly available LLM (ChatGPT) on comments written by real users on real YouTube channels and evaluate the extent to which these ``synthetic commenters'' can reproduce the types of comments written by humans. Our analyses evaluate the performance of these fine-tuned models across different sizes of the training data, as well as comparing models fine-tuned solely on comments written on conservative videos against those fine-tuned solely on comments written on liberal videos, both from US-oriented YouTube channels.
YouTube's Algorithm is Optimized for Rabbit Holes, Not Echo Chambers
with Vaibhav Gupta, Megan Brown, Angela Lai, Jonathan Nagler and Joshua Tucker (2024)
Working Paper
Abstract: While popular wisdom holds that online recommendation algorithms contribute to polarization in our political information environments, empirical scholarly research has largely concluded that these algorithms have a minimal impact on the narrowness of online content consumption. This conclusion presents something of a puzzle about what, if anything, these highly touted trade secret algorithms actually do. In this paper, we analyze three distinct sources of YouTube data collected in the fall of 2020 to address this question. We calculate: (1) the proportion of recommendations that are in the same category as the currently watched video, (2) the proportion of subsequent videos watched by users that are in the same category as the preceding video, and (3) the likelihood of a user selecting a recommendation from the same category as the current video. Our findings reveal that recommendation algorithms exhibit a tendency towards serial correlation. Moreover, user preferences also show strong serial correlation in the videos they watch. While these patterns are inconsistent with an echo chamber dynamic -- consistent with recent research -- they do suggest a "rabbit hole" dynamic in which users are recommended content similar to what they are currently watching. This study contributes to a deeper understanding of how both recommendation algorithms and user behavior collectively influence online information environments.
The Vibes are Off: Did Elon Musk Push Academics off Twitter?
with Kevin Munger (2024)
Conditionally accepted at POP
Abstract: Twitter has been a prominent forum for academics communicating online, both among themselves and with policymakers or the broader public. Elon Musk's take-over of the company brought sweeping change to many aspects of the platform, including the public access to its data; Twitter's approach to censorship and mis/disinformation; and tweaks to the affordances of the platform. In this letter, we take up a narrower empirical question: what did Elon Musk’s takeover of the platform mean for this academic ecosystem? Using a snowball sample of more than 15,700 academic accounts from the fields of economics, political science, sociology, and psychology, we show that academics in these fields reduced their ``engagement'' with the platform, measured with either the number of active accounts (i.e., those registering any behavior on a given day) or the number of tweets written (including original tweets, replies, retweets, or quote tweets). We further test whether this drop-off in engagement differed by account type, finding that verified users were significantly more likely to reduce their production of content (i.e., writing new tweets or quoting others' tweets), but not their engagement with the platform writ large (i.e., retweeting or replying to others’ content).
(I)PE
What You See Out Your Back Door: How Political Beliefs Respond to Local Trade Shocks
(2019)
Job Market Paper
Abstract: What effect does trade have on American politics? Until recently, the answer to this question was "not much". Trade has been viewed as a low-salience issue since the 1970s -- its once-important status among the policy dimensions superseded by issues such as abortion, gay rights, and gun control. However, recent research and the election of Donald Trump has thrust trade back into the spotlight. In this paper, I document a causal relationship between localized trade shocks and individual policy preferences, highlighting the importance of sociotropic concerns at the local level. In addition, I show that policy preferences on free trade are bundled with opinions on immigrants and US global leadership, assembling to create a nativist response to trade-related economic shocks. These findings further our understanding of how free trade's unequal economic consequences explain variation in the political economy of trade.
The Millennium Development Goals and Education: Accountability and Substitution in Global Assessment
with James R. Hollyer, James Raymond Vreeland, and Peter B. Rosendorff (2019)
International Organization
Abstract: Precise international metrics and assessments may induce governments to alter policies in pursuit of more favorable assessments according to these metrics. In this paper, we explore a secondary effect of global performance assessments (GPAs): Insofar as governments have finite resources and make trade-offs in public goods investments, a GPA that precisely targets the provision of a particular public good may cause governments to substitute away from the provision of other, related, public goods. We argue that both the main effect of the GPA (on the measured public good) and this substitution effect vary systematically based on the domestic political institutions and informational environments of targeted states. Specifically, we contend that both the main and substitution effects of GPAs should be largest for governments that are least accountable (opaque and non-democratic) and should be smallest for those that are most accountable. We test these claims using data on primary and secondary enrollment rates across 114 countries. We find that countries substitute toward primary (which is targeted by the MDGs) and away from secondary (which is not), and that these effects are mitigated as accountability rises.
Decompensating Domestically: The Political Economy of Anti-Globalism
with Layna Mosley, Tom Pepinsky, and B. Peter Rosendorff (2019)
Journal of European Public Policy
Abstract: The rise of populism across advanced industrial countries presents a challenge to the institutions and norms that make up the current global order and threatens to undo the global system that has enabled decades of free trade and investment. We outline in this paper a domestic political economy account of the contemporary crisis of the global order, rooted in disenchantment with the redistributive bargain between globalization’s winners and losers. We present individual and local-level evidence that is consistent with this account, first documenting the decline of the embedded liberal compromise over the past 40 years in Europe, and then providing individual-level evidence from the United States of growing protectionism and xenophobia in response to import exposure, particularly among respondents whose occupational profile is most risk-exposed.
The China Shock for Political Science: Describing the Limits of Causal Identification
(2021)
Working Paper
Abstract: Since it's initial rounds on the working paper circuits of 2010, Autor, Dorn and Hanson (2013) has promised an unprecedented opportunity to assess important theories of economics, international political economy, and politics. Specifically, through a careful accounting that mapped changes in Chinese imports to commuting zone-level labor markets, combined with a two-period difference-in-differences strategy and an instrumental variable robustness check, the "China Shock" claimed to capture exogenous variation in import competition. Over the course of the ensuing decade, a cottage industry of empirical research flowered that used the China Shock to identify the causal effect of import competition on everything from crime to military enlistment. However, the speed of this proliferating research often meant that careful attention to the identifying assumptions required to transplant the China Shock to new empirical contexts was lacking. In this paper, I define the China Shock for political scientists, discussing the opportunities it provides for exciting new work and circumscribing the limits of its applicability. I propose an alternative implementation of the China Shock that more tightly focuses the source of identifying variation on the date of Chinese accession to the WTO and implements a generalized difference-in-differences specification to overcome several (but not all) of the threats to identification inherent in existing methods.
Anti-Globalization Sentiment: Exposure and Immobility
with B. Peter Rosendorff (2024)
American Journal of Political Science
Abstract: Individuals with heightened labor market insecurity express more protectionist, xenophobic and isolationist sentiment. We construct a novel measure of labor market insecurity that combines an individual's industry-based exposure to import competition with an occupation-based measure of job immobility. Immobility captures the similarity of an individual's job to others in the economy, weighted by their prevalence. The holder of a job that is dissimilar to others in the industry or in the state experiences more anxiety regarding their labor market prospects in the face of a globalization shock, and is more likely to express anti-globalization sentiment.
Political Geography and Firm Strategies: How Electoral Competition Influences Local Job Creation
with Hye Young You (2024)
The Journal of Politics
Abstract: Among the many promises made by politicians to their constituents, job creation is universal. Most jobs are created in the private sector and the jobs that firms provide are crucial to a politician’s ability to successfully deliver on their campaign promises. Do firms strategically use job creation to exert influence? We take this question to the universe of firms in the United States between 1997 and 2018, linking each subsidiary to a congressional district. We find that firms open subsidiaries in more competitive districts, suggesting that firms employ politicians’ constituents as a political strategy to build ties with vulnerable legislators. Unlike other tools of political influence available to a firm, such as campaign contributions and lobbying, job creation is constrained by geography and time. We show that these patterns exist only for firms in geographically flexible industries and that local job creation increases a politician’s electoral survival.
Free Trade Attitudes and the Vote
with Todd Hertling (2024)
Working paper
Abstract: Does exposure to free trade's negative consequences cause Americans to be more protectionist, and do those trade attitudes in turn influence their voting behavior? The field of international political economy (IPE) has long posited that economic self-interest governs individuals' policy preferences in a theoretical framework known as open economy politics (OEP). Yet we know from individual-level data from a range of other subfields that political attitudes are influenced by far more than simple self-interest. We devise an empirical confrontation between the rational actor assumption undergirding the OEP framework and an alternative set of theories of political behavior which we combine under the umbrella term of partisan motivated reasoning (PMR). This study contributes to the field by combining individual-level panel data from the Views of Electoral Research (VOTER) study with firm-level measures of trade-related layoffs, derived from applications to the federal Trade Adjustment Assistance (TAA) program. We causally link exposure to free trade's negative consequences to two outcomes of interest to the political science literature: opinions on free trade policies, and vote choice for president in the 2012, 2016, and 2020 U.S. presidential elections. We demonstrate that---while protectionist attitudes increase in response to increasing exposure to trade-related layoffs---these changes have very little, if any, relationship to their voting behavior, even in the 2016 U.S. presidential election, which was characterized by an especially high national salience for the issue of trade policy.
Signals of Doom: Wildfire Smoke, Climate Change, and Media Coverage
with Guilherme Fasoline and Sofia Berrospi (2024)
Working paper
Abstract: What are the consequences of wildfire smoke pollution for climate change attitudes? As the United States faces an alarming increase in air pollution related to wildfires, there is a growing academic and policy interest in understanding how these events influence public attitudes and support toward climate policies. Prior research assumes that direct experience with climate hazards leads individuals to update their attitudes and preferences, yet the results are mixed and inconsistent. We argue that this inconsistency partly arises from overlooking the role of media in shaping climate change opinions, particularly considering the cognitive effort required for individuals to assess the complexity of climate change and its effects. We theorize that media coverage of wildfires will influence public concern about climate change, with varying effects across U.S. regions. Our study, utilizing individual-level panel data, geocoded wildfire smoke data, and local media coverage, reveals that direct exposure to climate hazards does not increase climate change concern. Rather, individuals adjust their levels of concern in response to media coverage, particularly in liberal areas where emphasis on the climate-wildfire connection is prominent. In contrast, conservative media coverage tends to diminish public concern about climate change. Taken together, these findings underscore the role of media in shaping public attitudes toward climate change and highlight the potential for wildfires to exacerbate political polarization on the issue.
Climate change and the shadow of the future
with Sofia Berrospi and Guilherme Fasolin (2024)
Working paper
Abstract: The consequences of climate change are becoming more tangible and severe over time. Young people are the ones closer to experiencing these imminent consequences. Despite this, age has been overlooked as a significant explanatory variable in the literature on climate change opinions. This article seeks to fill this gap. We synthesize different studies across psychology, social sciences, and biology to generate three versions of a standard rational actor model of a future-discounting agent whose shadow of the future is endogenous to their age. We then evaluate our expectations using data from the Climate Change in the American Mind survey spanning from 2008 to 2022, encompassing responses from over 30,000 participants. Our descriptive results mainly suggest a negative relationship between age and concern, but we also find evidence of a curvilinear relationship. We disentangle age and cohort effects and find no significant cohort effects. In addition, we provide evidence regarding risk perceptions and the role of the information environment. Older participants exhibit heightened concern for the harm climate change may cause them personally compared to the harm it may have on the United States, developing nations, or future generations. They also anticipate climate-related harm to manifest sooner than younger participants. Finally, we find that younger individuals are not inherently more informed about climate change and that self-reported media exposure increases with age. While merely descriptive, our findings suggest scholars of climate change and policy-makers should consider age as an important variable in explaining climate change public opinion.
Yellin' at Yellen: Hostile Sexism in the Federal Reserve Congressional Hearings
with Nicolo Fraccaroli and Andreas Kern (2024)
The Journal of Politics
Abstract: How prevalent is hostile sexism among U.S. politicians? We analyze the transcripts of every congressional hearing attended by the chair of the U.S. Federal Reserve from 2001 to 2020 using Janet Yellen as a bundled treatment to provide carefully identified evidence of sexism. We find that legislators who interacted with both Yellen and at least one other male Fed chair over this period interrupt Yellen more and interact with her using more aggressive language. Furthermore, we show that having a daughter reduces a legislator's hostility toward Yellen. These results are not a function of differences in either the content Yellen discusses in these hearings (measured with topic models) or her tone of communication. Our results provide carefully identified evidence of hostile sexism among both male and female legislators from both sides of the aisle when confronted with a female in a novel position of power, supporting a rich literature on gender roles and forms of sexism.
Public Opinion & Political Behavior
Objective Facts and Elite Cues: Partisan Responses to Covid-19
with Da In Diana Lee (2021)
The Journal of Politics
Abstract: American politics scholars disagree on the extent to which voters use policy information to evaluate politicians, versus relying on partisan cues to evaluate policies. We demonstrate the coexistence of both of these perspectives by studying the degree to which objective facts (measured with local Covid-19 cases) and partisan cues (measured with President Trump’s tweets about the virus) influence differences in the social distancing behaviors in Democrat and Republican counties in 2020. We find that both factors play an important role in social distancing, but that the relative importance between cues and facts favors the latter. Furthermore, the importance of these signals declines over time, suggesting a crucial but underappreciated dynamic of how partisan positions evolve in a Bayesian framework.
Flight to Safety: Covid-Induced Changes in the Intensity of Status Quo Preference and Voting Behavior
with Daniel Honig (2021)
The American Political Science Review
Abstract: We posit that anxiety alters the intensity of voters’ preference for the status quo, inducing a “flight to safety” towards establishment candidates. Leveraging the outbreak of the novel coronavirus across space and time during the Democratic primary election of 2020, we identify a causal effect of the outbreak on voting, with Biden benefiting between 7 and 15 percentage points at Sanders’ expense. A survey experiment in which participants exposed to an anxiety-inducing prompt choose the less disruptive hypothetical candidate provides further evidence of our theorized flight to safety among US-based respondents. Evidence from 2020 French municipal & US House primary elections suggests a Covid-induced flight to safety benefits mainstream candidates across a range of settings. We conclude that anxiety stimulates voter preference for status quo preserving candidates – incumbents and non-incumbents alike.
The Correlates of White Nationalism in 2021
with Patrick J. Egan and Joshua A. Tucker (2024)
R&R at Nature
Abstract: In recent years, white nationalism has become a more visibly violent force in American public life, representing a grave threat to the foundations of our multiracial and multicultural democracy. In this paper, we present results from a first-ever survey measuring support for white nationalism, conducted in 2021 with a nationally representative sample of 3,225 non-Hispanic white U.S. adults. Using techniques designed to reduce measurement error and social desirability bias, we find that 6.7% of white adults endorse a short statement explaining white nationalist beliefs. We examine the correlates of this support, finding that it is most pronounced among young (age 18-29) white men, whites with less education and lower incomes, and white conservatives and Republicans. We then provide descriptive evidence of three dominant theoretical frameworks for understanding radicalization: strain theory, structuralist theory, and social movement theory, finding that whites who live in places under societal stress--as indicated by high levels of poverty, unemployment, and deaths from substance use--are also more likely to support white nationalist ideas, as are whites who report more recent personal hardships. White nationalistic beliefs are also more prevalent among white people whose social worlds are largely online. But contrary to expectations, neither levels nor change in the share of local population that is white is associated with support for white nationalism.
Political Methodology
Local Instruments, Global Extrapolation: External Validity of the Labor Supply-Fertility Local Average Treatment Effect
with Rajeev Dehejia, Cristian Pop-Eleches, and Cyrus Samii (2017)
Journal of Labor Economics
Abstract: We investigate the external validity of local average treatment effects (LATEs), specifically Angrist and Evans’ (1998) use of same sex of the two first children as an instrumental variable for the effect of fertility on labor supply. We estimate their specification in 139 country-year censuses using Integrated Public Use Micro Sample International data. We compare each country-year's actual LATE to the extrapolated LATE from other country-years. We find that, with a sufficiently large reference sample, we extrapolate the treatment effect reasonably well, but the degree of accuracy depends on the extent of covariate similarity between the target and reference settings.
Testing Social Science Network Theories with Online Network Data: An Evaluation of External Validity
with Jennifer M. Larson (2017)
American Political Science Review
Abstract: To answer questions about the origins and outcomes of collective action, political scientists increasingly turn to datasets with social network information culled from online sources. However, a fundamental question of external validity remains untested: are the relationships measured between a person and her online peers informative of the kind of offline, "real-world" relationships to which network theories typically speak? This article offers the first direct comparison of the nature and consequences of online and offline social ties, using data collected via a novel network elicitation technique in an experimental setting. We document strong, robust similarity between online and offline relationships. This parity is not driven by shared identity of online and offline ties, but a shared nature of relationships in both domains. Our results affirm that online social tie data offer great promise for testing long-standing theories in the social sciences about the role of social networks.
BARP: Improving Mister P using Bayesian Additive Regression Trees
(2019)
American Political Science Review Letters
Abstract: Multilevel regression and post-stratification (MRP) is the current gold standard for extrapolating opinion data from nationally representative surveys to smaller geographic units. However, innovations in non-parametric machine learning methods can further improve the researcher's ability to extrapolate opinion data to a geographic unit of interest. I test an ensemble of machine learning algorithms and find that there is room for substantial improvement on the multilevel model via more sophisicated methods of regularization. I propose a modified version of MRP that replaces the multilevel model with a non-parametric approach called Bayesian Additive Regression Trees (BART or, when combined with post-stratification, BARP). I compare both methods across a number of data contexts, demonstrating the benefits of applying more powerful regularization methods to extrapolate opinion data to target geographical units. I provide an R package that implements the BARP method.
Corrigendum: The replication materials contain a mistake that inaccurately inflated the errors associated with MRP, exaggerating the improvements associated with BARP. Correcting this error yields a more modest comparison, albeit one in which BARP still outperforms MRP, especially in smaller samples. The corrigendum is forthcoming and will be linked here when published.
Geographic Boundaries and Local Economic Conditions Matter for Views of the Economy
with Jan Zilinsky (2022)
Political Analysis
Abstract: The link between objective facts and politically-relevant beliefs is an essential mechanism for democratic accountability. Yet the bulk of empirical work on this topic measures objective facts at whatever geographic units are readily available. In this letter, we investigate the implications of these largely arbitrary choices for predicting individual-level opinions. We show that varying the geographic resolution – namely aggregating economic data to different geographic units – influences the strength of the relationship between economic evaluations and local economic conditions. A systematic investigation of over 1.7 million subjective economic evaluations reveals that reliance on unemployment insurance is the best predictor of economic evaluations, especially when aggregated at the commuting zone or media market level. Our findings underscore the importance of researcher attention to a well-known but under-appreciated statistical challenge with observational data: the modifiable areal unit problem (MAUP). And our methods provide an example of how applied scholars might investigate these implications for their own research going forward.
Economic Evaluations and Partisan Faultfinding: When are Americans Most Likely to Answer Survey Questions Honestly?
with Jan Zilinsky (2024)
R&R at PSRM
Abstract: This paper introduces a simple framework for understanding which survey questions are more likely elicit political identity-influenced responses. We use daily data from Gallup to test which survey self-reports exhibit more or less susceptibility to politicization, finding the highest likelihood of politicization for macro-social questions. Conversely, we show that self-reported assessments of personal finances are less sensitive to partisan motivated responding. Taken together, our results uncover scope conditions for how to interpret self-reported views of the economy, and argue that measures of public opinion which have not yet been strongly politicized are better proxies for capturing the underlying welfare of the public.
Synthetic Replacements for Human Survey Data? The Perils of Large Language Models
with Joshua D. Clinton, Cassy Dorff, Brenton Kenkel and Jennifer M. Larson (2024)
Political Analysis
Abstract: Large language models (LLMs) offer new research possibilities for social scientists, but their potential as “synthetic data” is still largely unknown. In this paper, we investigate how accurately the popular LLM ChatGPT can recover public opinion, prompting the LLM to adopt different “personas” and then provide feeling thermometer scores for 11 sociopolitical groups. The average scores generated by ChatGPT correspond closely to the averages in our baseline survey, the 2016–2020 American National Election Study (ANES). Nevertheless, sampling by ChatGPT is not reliable for statistical inference: there is less variation in responses than in the real surveys, and regression coefficients often differ significantly from equivalent estimates obtained using ANES data. We also document how the distribution of synthetic responses varies with minor changes in prompt wording, and we show how the same prompt yields significantly different results over a 3-month period. Altogether, our findings raise serious concerns about the quality, reliability, and reproducibility of synthetic survey data generated by LLMs.