Facilitation of Research Networks & Best Practices
The Importance of Understanding Scope in Social Science Interventions
Social scientists have always studied human behavior and society, but in the last 50 years, they have increasingly turned to helping solve significant social problems by interventions and policies. When evaluating a possible intervention, social scientists look at the outcomes it leads to, its costs, and how easily and affordably it can be enlarged to include many people—its scale. IPR economist Jonathan Guryan, with the University of Chicago’s Monica Bhatt, Jens Ludwig, and Anuj Shah argue in a working paper that another aspect of an intervention’s effectiveness to assess is its “scope.” By scope, they mean how much an intervention can change a large share of the decisions that affect a person’s outcomes. A particular intervention might have a large impact on one decision an individual makes, but many decisions go into changing a given outcome. If one examined intervention shows a large effect, it might seem like an attractive policy change; however, the intervention might make little difference when considered as part of the whole range of decisions people make that produce improvements. The authors present a framework to examine policies that pays special attention to this range of decision making. Using it would enable social scientists to focus more on scope and to determine whether putting together multiple small-impact policies could result in large social changes. Guryan is the Lawyer Taylor Professor of Education and Social Policy.
Educational Research in the COVID Era
Larry Hedges and Elizabeth Tipton, IPR statisticians and co-directors of the STEPP Center, address the difficulties, due to the COVID-19 pandemic, facing educational researchers with experiments in the field. In a working paper, they systematically consider possible strategies for researchers at different stages of conducting funded randomized or longitudinal studies. These studies, meant to assess how well certain educational interventions work, cannot be conducted as planned in schools that closed overnight in March 2020
and are often still conducting classes online. Drawing on their own and others’ research, and their own experience, which includes running the Cluster-Randomized Trials Institute at IPR, Hedges and Tipton present a set of questions and options to assist the researchers—and their grant program officers—to think through next steps. They begin with the principles of quality in educational research found in the Institute of Education Sciences (IES) Standards of Excellence in Education Research (SEER). They start by asking researchers to consider whether a study should continue. If the answer is yes, then the authors offer four strategies to alter the design, methodology, or even focus of the study to address issues that will likely arise. Above all, they urge researchers tackling great uncertainties to base their decisions on scientific experience, practice, and judgment. Hedges is Board of Trustees Professor of Statistics and Education and Social Policy.
Why Race Should Remain a Factor in Clinical Decision Making
When physicians and other clinicians make decisions about how to treat a patient, they should consider all factors—such as health history, test results, demographic, and socioeconomic attributes—that predict a patient’s health risks. They should group patients according to their characteristics to assess the risks they run and then treat patients accordingly. Knowing more about patients allows more accurate predictions of risks and better decision making, IPR economist Charles F. Manski points out in a perspective published in Health Economics. In the last few years, however, researchers have reconsidered the use of race as one of the characteristics in assessments of patients’ risks of disease, and some have argued that race should no longer be included. Manski states that advocates for removing race as a diagnostic consideration have not studied how it would affect the quality of treatment decisions. He subsequently argues that using a “patient-centered perspective” leads to the conclusion that “race-free risk assessment would harm patients of all races.” He identifies and argues against four claims in support of race-free assessment: (1) Race is a social, not biological, concept; (2) race should not be considered if there is no established causal link between race and the illness; (3) using race may perpetuate or worsen race-based health inequities; and (4) many patients are offended by the use of race in risk assessment. He notes that although people may misuse race as a variable in the algorithms employed in risk assessment, race should be included in clinical decisions to give patients the best possible treatment appropriate to them as individuals—unless there is evidence that the social benefit of removing race as a factor in decision making is larger than the harm it causes to patient care. Manski is the Board of Trustees Professor of Economics.
The Case for Teaching Preregistration in Graduate School
Preregistration—the public posting of research plans before a study is completed—is a new tool to address the issue of replicating research in psychological science. In preregistration, the researcher documents the hypotheses and methods before data collection or analysis and saves them in an unchangeable format. Psychologist and IPR associate Jennifer Tackett and three of her graduate students document a graduate workshop course on preregistration and make recommendations based on their experience in Canadian Psychology. The two main goals of the course were (1) to teach students to think critically about conducting research in psychological science and about how to draw conclusions, and (2) to give them hands-on experience in improving their methods and practices. As part of the workshop, students wrote preregistrations for their own projects and contributed to creating a preregistration template as they considered the challenges of unusual types of research such as archival and nonexperimental analysis. Participants agreed that one of the most useful outcomes of the course was a detailed preregistration checklist, which is available online. The authors note that other areas, in and beyond psychology, might benefit from a similar course.