Improvements to Experimental Design & Quality
How to Measure Inequality in Small-Group Discussion
In any group working together, such as a jury, some people talk more than others. This inequality may promote efficiency, but sometimes means that some people, or certain kinds of people, have been ignored. Court opinions on jury size have discussed inequality in talk, with some scholars telling the courts that smaller groups, while offering less diversity in membership, are more egalitarian than larger ones. Is this true? Are smaller juries “better"? In a recent article, law professor, psychologist, and IPR associate Shari Seidman Diamond and her colleagues Mary R. Rose and Dan Powers question this conclusion because, the authors note, there are problems 
with measuring inequality in small groups. They apply three commonly used metrics to juries to evaluate which is most useful for comparing inequality in small groups. Using four highly realistic datasets from juries that deliberated, either in real trials or in experiments, the researchers tested the measures by counting each time someone new speaks and the number of words spoken by each juror. Diamond and her co-authors find that all three tested measures of inequality correlate with the number of words and turns of speech, but some measures falsely portray small groups as more egalitarian than they are. The authors show that a metric known as the index of concentration is the most useful in demonstrating the level of equality across small groups of differing sizes, and they urge more research into applying it. From a policy perspective, this study helps to shed light on what the best size for an equitable jury might be. Diamond is the Howard J. Trienens Professor of Law.
Designing a Five-Question Women's Agency Index
Measuring women’s agency—their ability to make choices and act on them—is key to research and policy related to gender equality. Although long survey measures of women’s agency exist, IPR economist Seema Jayachandran and her colleagues developed a shorter, five-question measure. They used a new method that combines machine learning and semi-structured interviews, which they tried in Haryana, India, and explain in an IPR working paper. The researchers collected 63 possible questions from existing surveys, such as the Demographic and Health Surveys and the Relative Autonomy Index. They also developed an interview guide that covered six areas of women’s agency: education, fertility, mobility, health, employment, and household spending. The survey and interviews were administered in Haryana, an area the researchers chose because of its large gender gaps, to 209 women from 21 villages from February–May 2019. To identify the best survey questions, the researchers used statistical analyses based on machine learning to eliminate all but five questions. These best questions were selected because the answers to them strongly predict the qualitative score obtained from the interviews. The top questions asked about a woman’s role in specific buying decisions, her mobility, and her children’s healthcare. The five-question module is a valuable new resource for measuring women’s agency and the new method may contribute to the development of a valid measure for other places and other complex concepts.
Perspectives From Transgender and Gender Diverse People on Asking About Gender
Research has increasingly focused on the inclusive measurement of transgender and gender diverse (TGD) people's gender identities, yet gaps still exist in understanding how these individuals prefer to be asked their gender in academic studies. In a study published in LGBT Health, professor of medical social sciences and IPR associate Brian Mustanski and his colleagues examine how TGD people desire their gender to be asked about and represented in research. In an online survey conducted between 2015 and 2017, the researchers asked 695 TGD people to provide written suggestions for how to ask about gender, and 314 gave suggestions. The participants were primarily White (75.7%) and between 16–73 years old. Three broad categories of responses emerged, including specific identities to include in response options, specific questions to ask about gender, and qualifiers or nuanced considerations, such as the option to check multiple boxes or offer a fill-in-the-blank question. Some participants also suggested a two-step method for asking about gender, such as asking the sex assigned at birth and then current gender, while others suggested there should be a question about gender followed by a specific question about whether participants were TGD. The researchers write that improving questions about gender is an important step to increasing accurate representation of TGD people in research, and future research is needed to continue evaluating these suggestions.
Better Designs for Replication Studies
How to check research results in medicine, psychology, education, behavioral economics, and other fields by replicating the experiments in the original studies has been a significant issue for close to 20 years. Current standards for research replication include conducting multiple studies, rather than a single study, to see if the results of the original research will be reproduced. In the Journal of the Royal Statistical Society: Series A, IPR education researcher and statistician Larry Hedges and former IPR postdoctoral fellow Jacob Schauer, now at Northwestern’s Feinberg School of Medicine, turn to how to design groups of studies that investigate replication. They ask how many studies and how many subjects per study are necessary to ensure statistically sound results that are cost-efficient and use a meta-analytic tool in their approach. The authors argue that “approximate replication” is valuable and more attainable than “exact replication,” which may be too strict a measure, and propose methods that can be used to create optimal designs that have sufficient statistical power for approximate replication. They describe the design for two forms of studies—hypothesis tests and variance component estimation—and then evaluate them by testing a current research design for replication used by the Many Labs Project, which examined a psychology study on the “retrospective gambler’s fallacy.” Hedges and Schauer conclude that the Project’s design had sufficient power to identify some larger differences between the studies being analyzed, but that other designs would have been less expensive and/or produced more precise estimates or higher-powered hypothesis tests. Hedges is Board of Trustees Professor of Statistics and Education and Social Policy.