Estimating the Accuracy of Jury Verdicts (WP-06-05)
Bruce D. Spencer
Many court cases are decided by juries, and it has long been known that juries sometimes make incorrect decisions. The accuracy of jury verdicts can be studied empirically and systematically. In the field of sample surveys, it is common to estimate sampling accuracy even though the true value is not observed—the key is to use replication. A similar approach can be used in methodology to estimate the accuracy of jury verdicts when “replications” such as second decisions on the cases are available. A recent study by the National Center for State Courts (NCSC) collected such replication data for criminal cases in four large U.S. metropolitan areas between 2000 and 2001. The levels of agreement between the judge’s verdict and jury’s verdict were only fair to poor, remarkably similar to the Kalven and Zeisel data collected in the 1950s. Estimates of jury accuracy can be developed from the replication data, although stronger assumptions are needed than for random sample surveys. Such assumptions are identified and the effects of failure of the assumptions are analyzed. Under some plausible conditions, the estimates tend to overstate jury accuracy.
Finer analyses of jury accuracy can be obtained with stronger models. Log-linear statistical models were fitted to the data to provide direct estimates of the differential accuracy between judge and jury. The models exploit a novel feature of the NCSC data: The inclusion of evaluations of the strength of evidence in the cases. The models are subject to error in specification, including the assumption of independence between the judge and the jury as well as imperfect measurement of strength of evidence. Thus, the estimates based on the model should be interpreted cautiously. The estimates from the model support the views that (1) juries were more likely than judges to acquit when the defendant was not guilty, and (2) judges were more likely than juries to convict when the defendant was guilty. The estimates do not support the view that the jury’s probability of convicting a not-guilty defendant is smaller than its probability of acquitting a guilty defendant. These findings, even taken with due caution, should not be generalized outside the NCSC study. Future replications of the NCSC study would be valuable and should be carried out over time and in other jurisdictions.