April 30, 2015
Judith Scott-Clayton, Columbia University and NBER, and Lauren Schudde, Columbia University
College attendance is a risky investment. But students may not recognize when they are at risk for failure, and financial aid introduces the possibility for moral hazard. Academic performance standards can serve three roles in this context: signaling expectations for success, providing incentives for increased student effort, and limiting financial losses. Such standards have existed in federal need-based aid programs for nearly 40 years in the form of Satisfactory Academic Progress (SAP) requirements, yet have received virtually no academic attention. In this paper, Scott-Clayton and Schudde sketch a simple model to illustrate not only student responses to standards but also the tradeoffs faced by a social planner weighing whether to set performance standards in the context of need-based aid. The researchers then use regression discontinuity and difference-in-difference designs to examine the consequences of SAP failure. In line with theoretical predictions, they find heterogeneous effects in the short term, with negative impacts on persistence but positive effects on grades for students who remain enrolled. After three years, the negative effects appear to dominate. Effects on credits attempted are 2-3 times as large as effects on credits earned, suggesting that standards increase the efficiency of aid expenditures. But it also appears to exacerbate inequality in higher education by pushing out low-performing low-income students faster than their equally low-performing, but higher-income peers.
Sarah R. Cohodes, Harvard University
Previous work on tracking high-achieving elementary and middle students in the U.S. has shown little impact on short-run test scores. Cohodes provides the first estimates of the long-run impacts of tracking using data from the Boston Public Schools' (BPS) program for high-achieving students, Advanced Work Class (AWC). AWC is an accelerated curriculum in 4th through 6th grades with dedicated classrooms. BPS offers AWC to students who score well on a 3rd grade exam. Using a fuzzy regression discontinuity approach, the researcher estimates the causal effect of AWC on standardized test scores, AP, SAT, high school graduation and college entrance. Like other programs for high-achieving students, AWC has little impact on test scores. However, it improves longer-term academic outcomes. AWC increases Algebra 1 enrollment by 8th grade, AP exam taking, especially in calculus, and college enrollment. It also has large positive effects on high school graduation for minority students. College enrollment increases are particularly large for elite institutions. One year of AWC attendance triples the rate of matriculation at a "most competitive" university. Using a multiple instrument strategy, Cohodes tests several potential channels for program effects to operate and find suggestive evidence that teacher effectiveness and math acceleration account for AWC effects, with little evidence that peer effects contribute to gains.
Hugh Macartney, Duke University and NBER; Robert McMillan, University of Toronto and NBER; and Uros Petronijevic, University of Toronto
This paper provides the first empirical analysis assessing the cost effectiveness of alternative education accountability systems, including those yet to be implemented. Macartney, McMillan, and Petronijevic set out a semi-parametric approach that uses exogenous variation in incentives to isolate the underlying effort response of teachers and schools. This response in hand, the researchers compute the average effort and dispersion of outcomes for a given cost under different accountability systems, including the most widespread those setting fixed and value-added targets. Varying the parameters of each type of scheme counterfactually allows the authors, in turn, to trace out associated performance frontiers on a comparable basis. To implement the approach, they use rich administrative data covering all public school students in North Carolina, showing that the introduction of the federal No Child Left Behind Act on top of the state's pre-existing accountability scheme led to marked improvements in scores, with students closer to the margin of passing experiencing larger gains. Differencing the score distributions post- versus pre-reform with respect to a continuous incentive strength measure uncovers the effort function used in the researchers' counterfactual approach. The types of estimates to emerge from this are new to the literature: the researchers show that there is a clear tradeoff between average performance and the tightness of the score distribution for both fixed and value-added schemes, and that the value-added frontier lies to the right of its fixed counterpart, associated with higher effort for a given cost. Further, a scheme that makes bonus payments student-specific can, for a given average effort level, reduce the variance of scores below that attainable under a regular value-added scheme. The analysis is relevant to the reform of existing accountability systems, with broader implications for using policy variables to improve education productivity.
Nicola Bianchi, Stanford University
In an effort to raise skills or promote equality, states sometimes engage in sweeping reforms that rapidly increase access to education for a significant share of their population. Such reforms are hard to evaluate because they may alter more than the outcomes of marginal students induced to enroll. They may change returns to skill, school quality, peer effects, and the educational choices of apparently inframarginal students (those who would have enrolled in the absence of the reform). Bianchi identifies such general equilibrium effects by examining a dramatic 1961 Italian reform increased university enrollment in science, technology, engineering, and math (STEM) fields by more than 200 percent in a few years. The peculiar features of the reform allows Bianchi to identify students who were unaffected, directly affected, and indirectly affected. The author identifies key channels through which the effects ran. Using data collected from tax returns and hand-written transcripts on more than 27,000 students, Bianchi shows that the direct effects of the reform were as intended: many more students enrolled and many more obtained degrees. However, the author also finds that those induced to enroll earned no more than students in earlier cohorts who were denied access to university. Bianchi reconciles these surprising results by showing that the education expansion reduced returns to skill and lowered university learning through congestion and peer effects. She also demonstrates that apparently inframarginal students were significantly affected: the most able of them abandoned STEM majors rather than accept lower returns and lower human capital.
Cassandra Hart, Elizabeth Friedmann, and Michael Hill, University of California, Davis
In this paper, Hart, Friedmann, and Hill use fixed effects analyses to estimate differences in student performance under online versus face-to-face course delivery formats in California's Community College system. On average, students have poorer outcomes in online courses, in terms of the likelihood of course completion; course completion with a passing grade; and receiving an A or B. These estimates are robust across estimation techniques, different groups of students, and different types of classes. Differences are especially acute in summer sessions, intersessions, non-transfereligible classes, and classes that enroll a smaller share of their students online. Differences in faculty characteristics only negligibly dampen the estimated relationships.
Markus Nagler, University of Munich; Marc Piopiunik, IFO; and Martin R. West, Harvard University
How do alternative job opportunities affect teacher quality? Nagler, Piopiunik, and West provide the first causal evidence on this question by exploiting business cycle conditions at career start as a source of exogenous variation in the outside options of potential teachers. Unlike prior research, the researchers directly assess teacher quality with value-added measures of impacts on student test scores, using administrative data on 33,000 teachers in Florida public schools. Consistent with a Roy model of occupational choice, teachers entering the profession during recessions are significantly more effective in raising student test scores. Results are not driven by differential attrition among teachers entering during recessions.