Economics of Field Experiments
April 10 and 11, 2015
Abhijit Banerjee; Sylvain Chassang, Princeton University; and Erik Snowberg, California Institute of Technology and NBER
Banerjee, Chassang, and Snowberg discuss possible decision theoretic foundations for experiment design and argue that a non-Bayesian minimax framework is best suited to capture the objectives of actual experimenters. The researchers use this framework to explore questions of optimal design, and external validity. They show that randomization and frequentist decision-making obtain naturally in internal decision making problems. However, external decision making remains inherently subjective. The authors embrace this conclusion and propose a framework for structured speculation that: (i) provides experimenters a formal way to express their beliefs about the external validity of their findings; (ii) exploits subjective beliefs in a robust way.
Duncan Simester, MIT
Marketing is a diverse field that draws from a rich array of disciplines and a broad assortment of empirical and theoretical methods. One of those disciplines is economics and one of the methods used to investigate economic questions is field experiments. The history of field experiments in the marketing literature is surprisingly long. Early examples include Curhan (1974) and Eskin and Baron (1977), who vary prices, newspaper advertising, and display variables in grocery stores. This chapter reviews the recent history of field experiments in marketing by identifying papers published in the last 20 years (between 1995 and 2014). Simester reports how the number of papers published has increased during this period, and evaluates different explanations for this increase. He then groups the papers into five topics and reviews the papers by topic. The chapter concludes by reflecting on the design of field experiments used in marketing, and proposing topics for future research.
Benjamin A. Olken, MIT and NBER; Rohini Pande, Harvard University and NBER; and Frederico Finan, University of California, Berkeley and NBER
Governments play a central role in facilitating economic development. Yet while economists have long emphasized the importance of government quality, much less attention has been given to the internal workings of the state and the individuals who provide the public service. Recently however, researchers have begun to explore the personnel economics of the state, and with the use of field experiments. Finan, Olken, and Pande summarize this nascent literature, by first documenting empirically the stark fact that in most countries throughout the world, public sector employees enjoy a significant wage premium over their private sector counterparts. Moreover, this wage gap is largest among low income countries, which tends to be precisely where governance issues are most severe. The researchers then examine the recent literature using field experiments to explore three aspects of the state-employee relationship selection, incentive structures, and monitoring. They conclude this survey with directions for future research.
Alan S. Gerber, Yale University and NBER, and Donald Green, Columbia University
In recent years the focus of empirical work in political science has begun to shift from description to an increasing emphasis on the credible estimation of causal effects. A key feature of this change has been the increasing prominence of experimental methods, and especially field experiments. In this chapter, Gerber and Green review the use of field experiments to study political participation. Although several important experiments address political phenomena other than voter participation (Bergan 2009; Butler and Broockman 2011; Butler and Nickerson 2011; Broockman 2013, 2014; Grose 2014), the literature measuring the effect of various interventions on voter turnout is the largest and most fully developed, and it provides a good illustration of how the use of field experiments in political science has proceeded. From an initial focus on the relative effects of different modes of communication, scholars began to explore how theoretical insights from social psychology and behavioral economics might be used to craft messages and how voter mobilization experiments could be employed to test the real world effects of theoretical claims. The existence of a large number of experimental turnout studies was essential to provide the background against which unusual and important results could be easily discerned. The researchers begin by describing the intellectual context of the modern emergence of field experiments to study voter turnout. They discuss the state of the literature on campaign effects and voter mobilization around the time of the re-introduction of field experimentation to study political behavior. The authors discuss some of the methodological reasons why this change represents an important advance over previous work. Their literature reviews focus on two broad areas of research: the effects of different modes of communication (face-to-face conversations, phone calls, mail, and mass media) and the effects of different messages. In the final section the researchers discuss some open questions and new directions for applications of field experiments to the application of field experiments to voter turnout and the study of political behavior more generally.
Pascaline Dupas, Stanford University and NBER, and Edward Miguel, University of California, Berkeley and NBER
Higher levels of health in developing countries could considerably improve wellbeing and possibly promote economic growth. The last decade has seen a surge in field experiments designed to understand the barriers that households and governments face in investing in health and how these barriers can be overcome, and to assess the impacts of subsequent health gains. In this chapter, Dupas and Miguel first discuss the methodological pitfalls that field experiments in the health sector are particularly susceptible to, then they review the evidence that rigorous field experiments have generated so far.
Karthik Muralidharan, University of California, San Diego and NBER
Roland G. Fryer, Jr., Harvard University and NBER
Judith Gueron, MDRC
Alain de Janvry and Elisabeth Sadoulet, University of California, Berkeley, and Tavneet Suri, MIT and NBER
Till M. von Wachter, University of California, Los Angeles and NBER, and Jesse Rothstein, University of California, Berkeley and NBER
Large scale social experiments were pioneered in labor economics, and have been used to study topics ranging from the effect of job training to incentives for job search to labor supply responses to taxation. Yet, many questions routinely asked in the context of social experiments in labor economics require going beyond random assignment. This includes questions pertaining to both internal and external validity, including endogenously observed outcomes, such as wages and hours; spillover effects; site effects; heterogeneity in treatment effects; multiple and hidden treatments; and the mechanisms producing treatment effects. In this chapter, Rothstein and von Wachter review approaches that address these design issues in the context of randomized control trials in labor. These approaches expand the range of questions that can be answered using experiments by combining experimental variation with econometric or theoretical assumptions. The researchers also discuss efforts to build the means of answering these questions into the ex ante design of experiments. Their discussion yields an overview of the expanding toolkit available to experimental researchers.
Uri Gneezy, University of California, San Diego, and Alex Imas, Carnegie Mellon University
Elizabeth Levy Paluck and Eldar Shafir, Princeton University
Marianne Bertrand, University of Chicago and NBER, and Esther Duflo
Dean Karlan, Yale University and NBER, and Rema Hanna, Harvard University and NBER
Anti-poverty programs that aim to help individuals establish a sustainable income source come in many varieties, and range from multi-faceted complex, even dynamic, interventions to simple cash transfers. The more complex and multi-faceted intervention pose a challenge for research. While straightforward to answer "whether" something works, it is far less straightforward to answer why it works. Hanna and Karlan discuss these challenges, and then focus on how careful experimentation, typically employing multiple-treatment arms, complemented with thoughtful data collection and analysis, can help overcome these obstacles. The researchers discuss issues along several dimensions, for example targeting of the program (and how that affects not just who participates but their level of engagement in the program), who implements the program, and the conditionality of transfers. Ultimately, with a stronger understanding of why programs work, or do not, one can make stronger policy prescriptions.
Jeffrey R. Kling, Congressional Budget Office and NBER; Jens Ludwig, University of Chicago and NBER; and William Congdon, ideas42
Policymakers and researchers are increasingly interested in using experimental methods to inform the design of social policy. In this chapter Congdon, Kling, Ludwig, and Mullainathan develop a framework that helps identify the different types of experimental designs that can be useful for informing policy decisions. The researchers distinguish between policy evaluations, which seek to understand the effectiveness of already designed policies, and mechanism experiments, which do not necessarily examine existing or even feasible policies but instead try to directly test the hypothesized causal mechanisms that are relevant for different policy interventions. The authors discuss how the selective use of mechanism experiments can complement policy evaluations and increase the amount of policy-relevant information that can be derived for a given research budget, and provide a number of examples from a range of social policy areas including health insurance, education policy, labor market policy, savings and retirement, housing, criminal justice, redistribution, and tax policy. Examples focus on the U.S. context.