Recent Performance Measurement and Rewards Research

Healthcare Markets and Regulation

Emergency Department Use Under the ACA
With the passage of the Affordable Care Act (ACA), newly insured patients have made increased use of hospital emergency departments. But little is known about any change in the location or type of emergency department service as a result of broader insurance coverage under the new law. In the Annals of Internal Medicine, IPR economist Matthew Notowidigdo, healthcare economist and IPR associate Craig Garthwaite, and their colleagues compare changes in emergency department use at the end of 2013 with the end of 2014, when Medicaid expansion under the ACA took effect. They find that before 2014, emergency department use was the same in Medicaid expansion and nonexpansion states. After 2014, hospitals in states that chose Medicaid expansion saw uninsured visits to the emergency department decrease by over 47 percent, while states that did not enroll in the expansion saw no decrease. In the expansion states, visits to the emergency department for nonemergency conditions fell by almost 44 percent among uninsured patients and increased by almost 130 percent among Medicaid patients. The researchers also discovered that the average travel time decreased by more than 6 percent among all Medicaid patients in expansion states, but not in the nonexpansion states.

Craig Garthwaite
Healthcare economist and IPR associate Craig Garthwaite
focuses on the effects of the Affordable Care Act,
examining changes in hospitals' uncompensated care
costs and variations in emergency department use before
and after Medicaid expansion.

The Economic impacts of Hospitalization
Fewer Americans are uninsured since the ACA’s passage. But even insured patients might face a big financial price if they are admitted to the hospital. To assess the economic impacts of hospital admissions, Notowidigdo, along with colleagues from the University of California, Santa Cruz, and MIT, closely examines the health and financial records of Californians who entered the hospital between 2003 and 2007. Excluding pregnancies and people who had been in the hospital in the last three years, the researchers find the average hospital stay lasted about four days and had a sticker price of $45,000—which is typically bargained down in later negotiations. Uninsured patients quickly fell into debt, according to the IPR working paper. Working-age adult insured patients were much less likely to carry major debt after being hospitalized, but their financial health did suffer. For example, insured adults between the ages of 50 and 64 saw their annual incomes fall by over $7,000, a decline of about 17 percent, and their out-of-pocket medical expenses rose by more than $1,000 a year following hospitalization. In contrast, elderly patients, who had health insurance through Medicare for medical expenses and had small labor-market earnings, appeared to suffer little or no economic setbacks from being hospitalized. These findings underline that insurance in the United States, unlike some European countries, rarely covers any decline in earnings.

Hospitals as Insurers of Last Resort
In the United States, hospitals are required to provide emergency medical care to the uninsured. How much does this cost them? In an IPR working paper, Notowidigdo, Garthwaite, and Tal Gross of Columbia University apply panel data methods and case studies from statewide Medicaid disenrollments, uncovering that hospitals’ uncompensated care costs increased in response to the size of the uninsured population. The findings suggest each additional uninsured person costs local hospitals $900 annually in uncompensated care. In addition, increases in the uninsured population lowered hospital profit margins, and hospitals could not pass along all increased costs to privately insured patients. For-profit hospitals were less affected by these factors, indicating that nonprofit hospitals hold a unique role as part of the social insurance system. Altogether, the results suggest that when uninsured individuals are unable to pay for emergency medical care, hospitals absorb much of the cost.

Measuring the ACA’s Impact on Hospitals
The ACA created the largest expansion in health insurance coverage since the creation of Medicare and Medicaid in the 1960s. Issues that the ACA was designed to address include the rising financial burden on both patients and hospitals that provide uncompensated care for uninsured people. Because little is known about new insurance enrollees’ hospitalization rates, or whether they were able to pay for medical services prior to the ACA taking effect, it is unclear how much uncompensated hospital care should have fallen as a result of the new legislation. In Health Affairs, healthcare economist and IPR associate David Dranove, Garthwaite, and Northwestern’s Christopher Ody provide the first national estimates of the decline in uncompensated care as a result of the ACA. Using Medicare Hospital Cost Reports, they find that uncompensated care costs in Medicaid expansion states—28 states plus the District of Columbia in the 2013–14 period studied—fell from 4.1 percentage points of operating costs to 3.1 percentage points of operating costs. However, in the nonexpansion states, uncompensated care costs remained about the same, at approximately 5.7 percentage points of hospitals’ operating costs. The authors calculate that if the 19 holdout states on ACA Medicaid expansion had instead adopted it, uncompensated care costs would have decreased by 1.6 percentage points—nearly 30 percent of their current level. Although uncompensated care has dropped in the Medicaid expansion states, some hospitals provide considerably more uncompensated care than others. Nonprofit hospitals have benefited especially from the expansion—a direct transfer of taxpayer dollars to hospitals’ profits. The researchers’ findings could help guide consideration of tax policies intended to promote charity care. Dranove is Walter J. McNerney Professor of Health Industry Management.

The High Prices of Pharmaceuticals
While expanded health insurance access, such as the ACA and Medicare Part D, can offer treatments that patients might not be able to afford otherwise, such expansions might also have negative effects. Dranove, Garthwaite, and Northwestern’s David Besanko investigate by developing a demand model where customers cannot afford expensive lifesaving treatments without insurance. They predict that in this setting insurance unambiguously increases the prices for these innovative treatments and in many cases decreases consumer surplus. Additionally, they predict that requiring insurers to cover a wide range of treatments in a single insurance bundle allows manufacturers of innovative products to set prices that exceed the value they create. The authors use the model to better understand the impact of the 2003 passage of Medicare Part D, which substantially expanded the number of seniors receiving drug coverage. They find that this insurance expansion raised prices in the oncology market and increased the probability that new products would be priced above the value they create.

Free Health Insurance in India
Since 2008, India’s public health insurance has covered the poorest quarter of the population. The Indian government is considering expanding coverage to include households above the poverty line. To create the most effective policy for nearly 700 million people, it is vital to understand how care-seeking behavior works with and without insurance. IPR economist Cynthia Kinnan and fellow researchers from the United States, England, and India are currently conducting a randomized controlled field experiment in Karnataka, India, to measure the effects of expanding the free inpatient public health insurance plan called Rashtriya Swasthya Bima Yojana (RSBY). RSBY is centrally funded, state-run, and provides hospitalization insurance to households below the poverty line. Some 11,000 households above the poverty line are participating in the study. The goal of the research is to quantify how the RSBY program benefits health and reduces poverty in the short and medium term.

Performance in Government & Nonprofits

The Perils of Pay-for-Performance
Government and nonprofit activities are often difficult, if not impossible, to measure and assign monetary value to in ways that are broadly acceptable. These difficulties prevent the adoption of incentive systems that align rewards with “performance.” IPR economist Burton Weisbrod is writing a book, under contract with Stanford University Press, to consider the unintended—but foreseeable and counter-productive—consequences of the rising tide of efforts to reward performance measures, such as standardized tests for schoolchildren and crime rates for police. These involve adopting measures that are both incomplete and biased and then rewarding them. Titled “The Perils of Pay-for-Performance: Why Strong Rewards in Government and Nonprofit Organizations Do Not Work,” the book will cover a wide array of public and nonprofit sector services, such as K–12 and higher education, hospitals, policing and jails, museums, charities, and the federal judiciary. For example, the debate surrounding No Child Left Behind (NCLB) has centered on whether the strong rewards and penalties for performance—and its increasing reliance on measurability—should be expanded to other areas of government and the nonprofit world. Weisbrod emphasizes how the forces at work in these systems cause strong rewards to be strategically “gamed,” so that it is not truly good performance that is rewarded, but measured performance—even when that is systematically overstated. He explains why the two concepts are in conflict—why, for example, larger rewards in public schools might improve test scores but not learning. Weisbrod directs IPR’s Seminar on Performance Measurement series, which features presentations by researchers studying healthcare, education, policing, and other social service industries. He is Cardiss Collins Professor of Economics.

The Performance of Earthquake Maps
In 2011, the 9.0 magnitude Tohoku earthquake and resulting tsunami killed more than 15,000 people and caused nearly $300 billion in damages. The shaking from the earthquake was significantly larger than Japan’s national hazard map had predicted. Such hazard-mapping failures, which can cause major damage in areas forecast to be relatively safe, prompted three Northwestern researchers—IPR statistician Bruce Spencer, geophysicist and IPR associate Seth Stein, and IPR graduate research assistant Edward Brooks—to assess the performance of earthquake hazard maps. Such maps are used around the world to make costly policy decisions for earthquake-resistant construction. To gauge the performance of the Japanese maps, the researchers created uniform and random maps of the Japanese earthquake hazards. The authors’ overall finding is that the Japanese national hazard maps are not performing as well as might be hoped. They consistently overpredict shaking—the expected shaking occurred at a much lower fraction of sites than predicted. At the same time, the national maps describe the observed shaking better than a uniform or randomized map. In other words, there are multiple and complex aspects to earthquake map prediction. In some metrics, the national maps do better—and in others worse—than the uniform or randomized maps. The scholars stress that seismologists need to know a lot more about how these maps work to improve them. Stein is William Deering Professor of Earth and Planetary Sciences.

Cash for Carbon: Does It Work?
Climate change is an enormous challenge, yet we have little evidence on what policies can mitigate it cost effectively. Reducing carbon emissions through forest conservation is widely considered cost effective, and thus it plays a key role in international climate policy. Various programs have been launched to slow down deforestation in developing countries, where most of it occurs today. IPR economist Seema Jayachandran and her colleagues conducted a randomized controlled trial to determine whether a program that paid Ugandan forest owners to conserve their forests led them to cut down fewer trees, and how the payment program’s benefits compare to its costs. Over two years, 121 villages in western Uganda participated in the experiment. In 60 of them, forest owners were paid to conserve their trees. Tree cover was measured with high-resolution satellite imagery before and at the end of the two-year trial. Forests in the villages where payments were made declined by 2–5 percent compared with 7–10 percent in the control villages during the study period. The researchers calculate the cost-effectiveness of the program in terms of averted carbon dioxide (CO2) emissions. In an IPR working paper, they estimate that for every 25 cents in payments, or 57 cents in total program costs, a ton of CO2 emissions due to deforestation was delayed. Using the accepted “social cost of carbon,” they conclude that the social benefit of the delayed carbon dioxide emissions is $1.11 per ton, or almost twice the 57-cent program cost. 

Matthew Notowidigdo
IPR economist Matthew Notowidigdo
(right) follows up with Rourke O'Brien
of the University of Wisconsin–Madison
after O'Brien's presentation on how
Medicaid leads to improved 
intergenerational economic mobility.

The Marginal Propensity to Consume
Ten years after an individual declares bankruptcy, the record of that bankruptcy is removed from his or her credit report, leading to an immediate increase in credit score. In an IPR working paperNotowidigdo and his co-authors examine the effects of this “bankruptcy flag” removal. Using a sample of 160,000 bankruptcy filers whose bankruptcy records were removed between 2004 and 2011, they determine that, in the year following flag removal, credit card limits increased by $780 and credit card balances increased by about $290. These findings point to a Marginal Propensity to Consume (MPC) out of liquidity—or the change in spending in response to the change in available assets—of 0.37. Focusing in on the years 2007–09—during the Great Recession—they identify a 20–30 percent increase in MPC. The results are consistent with typical models, where liquidity constraints become more important during recessions.

Combining Insights from Literature and Economics
Morton Schapiro, Northwestern University president, professor, and IPR economist, and Gary Saul Morson, Lawrence B. Dumas Professor of the Arts and Humanities and professor of Slavic languages and literatures, have been co-teaching extremely popular undergraduate courses on decision-making and alternative choices since 2011. The collaboration between the economist and the humanist has resulted in their second book, Cents and Sensibility (Princeton University Press, 2017). It breaks down the barriers between the academic bailiwicks of economics and the humanities to address issues ranging from the economics of the family to the development of poor nations. In one chapter, Schapiro and Morson discuss higher education, unpacking the well-known U.S. News & World Report college rankings. They examine the ways in which universities calculate and report statistics such as admission rate, incoming students’ standardized test scores, and alumni giving. How do the economist and humanist view the case of a university recording an alumni group’s $20,000 donation as 20,000 $1 donations to raise its alumni giving rate, and thus its overall ranking? Schapiro and Morson call on literary examples—such as portrayals of dishonesty in Tolstoy’s Anna Karenina—to explore the ethical implications of reporting potentially inaccurate statistics. Through their discussion of higher education and numerous other topics, the scholars argue that the study of literature offers economists ways to improve their models and predictions, as well as ways to shape more effective policies. At the same time, they underscore, examining literature through the lens of real-world economic problems can help to revitalize its study.

Accountability in Education

Improving Accountability Under ESSA
In a strategy paper for The Hamilton Project, its director, IPR economist Diane Whitmore Schanzenbach, and her co-authors apply some lessons from NCLB to the nation’s new education accountability law, the Every Student Succeeds Act (ESSA). ESSA gives states more freedom to create their education policies, but requires annual measurement of five indicators toward each state’s educational goals. Four of the five are defined, with the “fifth indicator” of “school quality or student success” left up to the states. It must, however, be evidence-based, systematically measurable, and related to improvements in student achievement and high school graduation. The authors recommend chronic absenteeism as their “fifth indicator.” More than six million students were chronically absent—defined as 15 days or more during a school year—in 2013–14. After analyzing how accountability indicators were measured and used under NCLB, the researchers conclude that chronic absenteeism is valid, reliable, comparable, and easy to measure. Rates of chronic absenteeism meaningfully differentiate between schools within each state and relate to school-wide measures of students’ achievement, growth, and graduation rates. The authors find evidence that reducing the rate of chronic absenteeism would likely increase student achievement and raise graduation rates.

No Child Left Behind and Obesity
In Education Finance and Policy, Schanzenbach, with Patricia Anderson of Dartmouth College and Kristin Butcher of Wellesley College, looks at how accountability pressure from the NCLB might have had the unintended consequence of affecting students’ body weight. They create a unique dataset that combines NCLB rules, test scores, and students’ body mass index. What is the connection? Schools might respond to the threat of sanctions under NCLB rules in a variety of ways. They might allocate time formerly used for physical activity to instructional activities, for example, taking time from recess or gym for more classwork and homework. Schools on the margin of NCLB sanctions have a 0.5 percentage point higher growth rate in their students being overweight. These results provide direct evidence that the NCLB accountability rules appear to have unintended adverse effects for students’ weight and, therefore, health.