Data Use, Quality & Cost in Policy Research
How Schools Are Chosen for Research
As education policymakers increasingly rely on evidence, researchers have conducted more cluster-randomized trials. But much work remains be done in reviewing those trials’ methodology, specifically how researchers choose which schools to include as what works for one school or district may not work for another. In a recent working paper, IPR statistician Beth Tipton and her co-authors examine 34 such trials to determine whether their samples are truly representative of particular populations of schools and students. They compare the sample data from those studies, funded by the Institute of Education Sciences and selected and evaluated by Tipton and her co-authors, to general population data from the U.S. Department of Education. They find that recruitment for studies is heavily dependent on pre-existing local relationships between researchers and the schools that are ultimately selected. Therefore, those schools skew, like the universities, larger in size and more urban than the population being studied. They find this poses major challenges to any generalizations drawn from such studies. The researchers recommend three major changes to recruitment: including the sample-collection methodology in the grant proposal, increasing training for sample collection, and establishing best practices for school recruitment.
Economics of Scaling Up
When researchers conduct randomized controlled trials (RCTs) of social programs, the hope is that smaller-scaled programs that appear promising in initial RCTs can then be implemented at a larger scale with a larger set of resources. But how can it be known whether a social program will be equally successful at scale without having been tested at scale? IPR economist Jonathan Guryan proposes a way to measure the economics of scaling up in an IPR working paper, along with Jonathan Davis, Kelly Hallberg, and Jens Ludwig of the University of Chicago. Their model focuses on one scale-up challenge specifically: For most social programs, labor is a key input, yet as social programs scale up and seek new hires, a search problem can arise. This results in a situation where, as programs are scaled, either the average cost of labor must go up or program quality will decline at fixed costs. For instance, a tutoring program that is being scaled up will eventually face a labor supply problem where the program will either need to start offering higher pay to attract high-quality tutors, or will have to accept lower-quality tutors at lower pay. While acknowledging that exact costs of scale-up cannot necessarily be known, Guryan and his co-authors show that it is possible to create and test a program at a smaller scale while still learning about the input supply curves, such as the labor supply curve, facing the firm at a much larger scale. This can be done by randomly sampling the inputs the provider would have hired if they operated at a larger scale. Highlighting the specific case of a Chicago tutoring program they are evaluating and trying to scale up, the researchers show how scale-up experiments can provide insights and confidence that cannot be derived from simply scaling up without an experiment at a modest scale. Guryan is the Lawyer Taylor Professor of Human Development and Social Policy.