Research News

Rethinking Measures of Socially Sensitive Issues

Results suggest how to improve studies using self-reported behaviors


Coffee cup
           The caffeine in a large cup of coffee qualifies as a "controlled substance" under current NCAA rules.

Drug and alcohol use among U.S. college students, particularly student-athletes, is often the source of much attention. Yet the measures researchers typically use to estimate the issue might be biased. An IPR working paper, coauthored by IPR political scientist James Druckman and forthcoming in Social Science Quarterly, suggests that survey methods using self-reports—in which participants take a survey without any subsequent verification as to the authenticity of their responses—might fail to provide accurate measures of socially sensitive issues, such as using banned substances.

Druckman
         James Druckman

Druckman, with Northwestern graduate student Mauro Gilli, Aarhus University (Denmark) postdoctoral fellow Joshua Robison, and the University of Arizona’s Samara Klar, offers new evidence indicating that existing research on student-athletes likely “understates usage” of alcohol and banned substances due to their fears of reprisal and social stigma.

Using a survey method known as a list experiment, the researchers gave more than 1,300 student-athletes a list of (relatively) socially acceptable activities they might or might not have participated in during college (e.g., “skipped a class because you felt so tired from a practice or a game”). The researchers asked how many items on the list the students had participated in, not which ones. This method eliminates the fear inherent to traditional self-reporting methods because students are not directly admitting to certain activities.

The control group received a list of four “socially acceptable” items, while a treatment group received a list with a fifth asking if they “[k]nowingly took a drug banned by the NCAA [National Collegiate Athletic Association] that may improve your athletic performance.” A second experiment used the same method, but it featured an item asking if student-athletes “[i]n the typical week during the past academic year, consumed more than five alcoholic drinks.” By comparing the mean responses between the control and treatment groups, the researchers could estimate the aggregate percentages of students who used banned substances or engaged in heavy drinking.

The differences in results were stark. The list experiment found 37 percent of respondents had knowingly taken banned drugs, compared with only 4.9 percent who admitted to it when directly asked in the survey. The list experiment revealed 46 percent consumed more than five drinks a week, compared with 2.4 percent who self-reported doing so.

Druckman and his colleagues warn against jumping to too many conclusions from the results about drug and alcohol use. Many of the student-athletes drinking might be over 21, for instance, and there could be confusion as to what constitutes a controlled substance, as the small amount of caffeine one ingests in drinking a large cup of coffee would qualify.

“We don’t know what banned substances are being used, who is drinking that much, and what they’re drinking,” Druckman said, adding that he suspects the study's findings on controlled substances stem from the caffeine issue. Future work, he suggested, is needed to pinpoint which substances are being abused and to what extent, including how much caffeine athletes are prone to consume. 

Still, the researchers demonstrate that an overreliance on self-reported data on a sensitive topic like this can be problematic; therefore, using other methods, such as a list experiment, is more likely to offer better data and insights. 

James Druckman is Payson S. Wild Professor of Political Science and IPR associate director and fellow. He also chairs IPR’s research program on Politics, Institutions and Public Policy. The working paper, “Measuring Drug and Alcohol Use Among College Student-Athletes” (WP-14-10) is available on IPR’s website and is forthcoming in Social Science Quarterly.