Facilitation of Research Networks & Best Practices
The Importance of Understanding Scope in Social Science Interventions
Social scientists have always studied human behavior and society, but in the last 50 years, they have increasingly turned to helping solve significant social problems by interventions and policies. When evaluating a possible intervention, social scientists look at the outcomes it leads to, its costs, and how easily and affordably it can be enlarged to include many people—its scale. IPR economist Jonathan Guryan, with the University of Chicago’s Monica Bhatt, Jens Ludwig, and Anuj Shah argue in a working paper that another aspect of an intervention’s effectiveness to assess is its “scope.” By scope, they mean how much an intervention can change a large share of the decisions that affect a person’s outcomes. A particular intervention might have a large impact on one decision an individual makes, but many decisions go into changing a given outcome. If one examined intervention shows a large effect, it might seem like an attractive policy change; however, the intervention might make little difference when considered as part of the whole range of decisions people make that produce improvements. The authors present a framework to examine policies that pays special attention to this range of decision making. Using it would enable social scientists to focus more on scope and to determine whether putting together multiple small-impact policies could result in large social changes. Guryan is the Lawyer Taylor Professor of Education and Social Policy.
Educational Research in the COVID Era
Larry Hedges and Elizabeth Tipton, IPR statisticians and co-directors of the STEPP Center, address the difficulties, due to the COVID-19 pandemic, facing educational researchers with experiments in the field. In a working paper, they systematically consider possible strategies for researchers at different stages of conducting funded randomized or longitudinal studies. These studies, meant to assess how well certain educational interventions work, cannot be conducted as planned in schools that closed overnight in March 2020 and are often still conducting classes online. Drawing on their own and others’ research, and their own experience, which includes running the Cluster-Randomized Trials Institute at IPR, Hedges and Tipton present a set of questions and options to assist the researchers—and their grant program officers—to think through next steps. They begin with the principles of quality in educational research found in the Institute of Education Sciences (IES) Standards of Excellence in Education Research (SEER). They start by asking researchers to consider whether a study should continue. If the answer is yes, then the authors offer four strategies to alter the design, methodology, or even focus of the study to address issues that will likely arise. Above all, they urge researchers tackling great uncertainties to base their decisions on scientific experience, practice, and judgment. Hedges is Board of Trustees Professor of Statistics and Education and Social Policy.
The Case for Teaching Preregistration in Graduate School
Preregistration—the public posting of research plans before a study is completed—is a new tool to address the issue of replicating research in psychological science. In preregistration, the researcher documents the hypotheses and methods before data collection or analysis and saves them in an unchangeable format. Psychologist and IPR associate Jennifer Tackett and three of her graduate students document a graduate workshop course on preregistration and make recommendations based on their experience in Canadian Psychology. The two main goals of the course were (1) to teach students to think critically about conducting research in psychological science and about how to draw conclusions, and (2) to give them hands-on experience in improving their methods and practices. As part of the workshop, students wrote preregistrations for their own projects and contributed to creating a preregistration template as they considered the challenges of unusual types of research such as archival and nonexperimental analysis. Participants agreed that one of the most useful outcomes of the course was a detailed preregistration checklist, which is available online. The authors note that other areas, in and beyond psychology, might benefit from a similar course.