Recent Performance Measurement and Rewards Research

Research Performance in Government and Nonprofits

The Perils of Pay for Performance

Government and nonprofit activities are often difficult, if not impossible, to measure and also to assign monetary value to. As a result, these difficulties prevent the adoption of systems that align rewards with “performance.” IPR economist Burton Weisbrod is writing a book under contract with Stanford University Press to consider the unintended—but foreseeable—consequences of the rising tide of efforts that often involve adopting incomplete and biased measures of “performance” and then rewarding it. Titled “The Perils of Pay-for-Performance: Why Strong Rewards in Public and Nonprofit Organizations Do Not Work,” the book will cover a wide array of public and nonprofit sector services, such as higher education, hospitals, policing, museums, private charities, the federal judiciary, and K–12 education. For example, the debate surrounding No Child Left Behind (NCLB) has centered on whether the strong rewards for performance under NCLB—and its increasing reliance on measurable performance—should be expanded to other areas of government and the nonprofit world. Weisbrod emphasizes how the forces at work in these systems cause strong rewards to be strategically “gamed” so that it is not truly good performance that is rewarded, but measured performance. He explains why the two concepts are systematically different—why, in short, larger rewards in public schools bring higher test scores but not greater learning. Weisbrod is John Evans Professor of Economics.

Invited Scholars Address Performance Measures

Over the year, the IPR Series on Performance Measurement and Rewards in the Public and Nonprofit Sectors welcomed three scholars: New York University sociologist Richard Arum presented follow-up research to his 2011 book, Academically Adrift: Limited Learning on College Campuses (University of Chicago Press) on January 29. He discussed the postcollege transitions of the same cohort of students, graduating in spring 2009 during a major economic crisis. On May 7, Henry Hansmann, Oscar M. Ruebhausen Professor of Law at Yale University, spoke about his work comparing the performance of companies owned primarily by charitable foundations with investor-owned companies. On November 5, IPR labor and education economist Kirabo Jackson shared research on evaluating teachers not just on test scores, but also on students’ noncognitive abilities, measured by outcomes like high school completion and college entrance exams.

Healthcare Markets and Regulation

The Cost of Electronic Medical Records

In 2009, Congress passed the Health Information Technology for Economic and Clinical Health Act (HITECH) that provides $27 billion in incentive payments to hospitals adopting electronic medical records (EMR). In an IPR working paper, healthcare economist and IPR associate David Dranove, Northwestern’s Craig Garthwaite and Christopher Ody, and Cornerstone Research’s Bingyang Li measure HITECH’s effectiveness in spurring EMR adoption. The researchers discover that EMR adoption rates markedly increased after HITECH’s passage, but note that EMR adoption was an ongoing trend before HITECH, and likely would have continued to grow without congressional intervention. They estimate that 77 percent of the hospitals in their survey would have adopted EMR anyway by 2013, only two years after hospitals reached this rate under HITECH—and without $27 billion in government spending. Moreover, a significant portion of HITECH’s funding went to early EMR adopters, hospitals that likely would have adopted the technology anyway. Yet, reducing incentives for early adopters could also delay adoption of new technologies, as hospitals would hold out for government money at a later date. Dranove and his colleagues’ work underscores the difficulty policymakers face when designing and providing incentives. Dranove is Walter J. McNerney Professor of Health Industry Management.

More Insurers Lower Healthcare Premiums

With just one year of data on health insurance marketplaces created by the Affordable Care Act (ACA), economists can glean information about the effect of competition on premiums. In a National Bureau of Economic Research (NBER) working paper, healthcare economist and IPR associate Leemore Dafny, with Ody and MIT’s Jonathan Gruber, examines the effect of insurer participation in the health insurance marketplaces established by the ACA on 2014 premiums. Using data from the 34 states with federally-facilitated marketplaces, Dafny and her colleagues find that if United Healthcare—the nation’s largest insurer—had participated in the first year of the marketplaces, the second- lowest-priced, silver premium, would have decreased by 5.4 percent on average. If all insurers active in each state’s individual insurance market in 2011 had participated, they estimate silver premiums would be slightly more than 11 percent lower, reducing federal subsidies by $1.7 billion. Dafny is Herman Smith Research Professor in Hospital and Health Services. 

Employment Lock and Health Insurance

Prior to the passage of the Affordable Care Act, health insurance was tightly linked to employment, as many Americans were only able to access it through their employer. This raises the possibility that some Americans were working solely for affordable health insurance, a phenomenon known as “employment lock.” In the Quarterly Journal of Economics, economists Matthew Notowidigdo, Garthwaite, and Columbia’s Tal Gross investigate the effect of public health insurance eligibility on the labor supply using data from a large public health insurance disenrollment in Tennessee. In 2005, approximately 170,000 Tennessee residents lost public insurance coverage under TennCare, the state’s Medicaid system. Following this statewide disenrollment, the researchers note that losing public insurance coverage prompted former TennCare recipients to search for jobs, become employed, and sign up for private health insurance coverage. The study results show that becoming eligible for public health insurance can have large effects on the labor market and that public health insurance also acts as a powerful work disincentive—results that might be applicable to the expansion of public insurance under the Affordable Care Act and were discussed in a Congressional Budget Office report.

Hospitals as Insurers of Last Resort

A combination of factors—an unfunded government mandate, charitable obligations, and medical ethics—require U.S. hospitals to provide emergency medical care to patients regardless of their health insurance status or ability to pay. Notowidigdo and his colleagues use previously confidential hospital financial data to study how these factors shape the relationship between health insurance and hospital uncompensated care costs, or medical care for which no payment is received. Using both across- and within-state variation in health insurance rates, they find that a 10 percentage point increase in a state’s share of the uninsured raises total hospital uncompensated-care costs by roughly $80 per person. These results are highly concentrated in private nonprofit hospitals with an emergency department, while for-profit hospitals and hospitals without an emergency department are largely unaffected. They use hospital-level panel data to show that uncompensated care costs fall sharply when a hospital permanently closes its emergency department, and that hospital closures increase the uncompensated-care costs of nearby hospitals. Taken together, the results reveal private nonprofit hospitals are insurers of last resort and can help with understanding the challenges of maintaining access to emergency medical treatment for individuals living in areas with low rates of health insurance.

Bequest Motives and Self-Insuring Late-Life Risks

Despite facing significant uncertainty about how long they will live and how much healthcare they might need in their later years, few retirees buy life annuities or long-term care insurance. Low rates of long-term care insurance coverage are typically seen as evidence that bequest, or inheritance, motives are not important to many, since not purchasing insurance puts one’s bequests at risk. However, research by economist and IPR associate Lee Lockwood suggests the opposite might be true. In a working paper, he observes that low rates of insurance coverage for long-term care—especially in combination with the slow rate at which many retirees draw down their wealth—are evidence that retirees do think about their inheritance. For those who value leaving an inheritance, which could be considered a luxury good, they would rather save the money they would have otherwise spent on premiums, essentially self-insuring. Incorporating estimations of life-cycle models, Lockwood finds that people without bequest motives—who want to consume all of their wealth—are less willing to make the consumption sacrifices to self-insure by saving than are those with bequest motives. 

Accountability Measures for Service Industries

Managerial Control and Performance Pay

In an IPR working paper, economist Kirabo Jackson and Henry Schneider of Cornell University are among the first to evaluate how managerial control might improve employee performance and compare it with performance pay. They studied the auto-repair industry, where performance pay is widespread. Their five-year study collected data on 108 mechanics, as well as their managers, customers, and repairs. To measure the effects of increased managerial control, the researchers distributed checklists in three randomly selected shops. Around one-third of the time, the mechanics filled them out, returning them to their supervisors. Jackson and Schneider discovered that when using checklists, the mechanics worked more hours and did more repairs. Shop revenue also increased by 20 percent. But when the experiment ended, the mechanics quit using the checklists—indicating that worker shirking was occurring. In another experiment gauging performance pay, the researchers observed mechanics who worked mainly on commission. A 6 percent increase in commissions led to a jump of 11.7 percent in shop revenue. Because the checklist method brought in more money than increasing commissions, the researchers concluded that managerial control is a viable means of improving worker outcomes and that, in some cases, it might even be more cost effective than raises. This research on performance pay could be considered when discussing policies to boost teacher effort. 

Measure of States’ Malpractice Environment

While the U.S. medical malpractice environment is the subject of much policy and research interest, law and finance professor Bernard Black, an IPR associate, and his colleagues argue that studies of this environment are lacking. In a 2014 Health Services Research article, the researchers seek to develop a composite measure of the malpractice environment in different states. They point to many studies that use a range of single indicators as measures of malpractice environment, such as the frequency of malpractice claims and lawyer density. They argue that these measures can be useful to test hypotheses about specific aspects of malpractice environments, but are often used as stand-ins for “a broader construct of ‘malpractice environment.’” Using data from multiple sources on lawyers, tort activity, and malpractice reforms, premiums, and payments, they created a composite of seven measures from previous studies. This composite measure accounted for more than 73 percent of the total variance in the seven indicators and demonstrated reasonable criterion validity. The researchers hope that this composite measure will be used as a complement to individual indicators to better understand and compare state malpractice environments. Black is Nicholas D. Chabraja Professor at Northwestern University School of Law and Kellogg School of Management.

Medical Malpractice Law and Health Outcomes

Climbing premiums for medical malpractice insurance have prompted many states to enact tort reforms—laws that deal with injuries to people and property—to mitigate the costs of medical malpractice litigation. However, little is known about how medical liability claims might affect the quality of care. In an IPR working paper, health and law scholar Michael Frakes and co-author Anupam Jena of Harvard University use two surveys to analyze the effects of medical malpractice law on several measures of inpatient and outpatient healthcare quality, including rates of avoidable hospitalization and of medical errors for mothers during childbirth. Their results reveal that damage caps have “at most, a modest role” in improving healthcare. In particular, caps on noneconomic damages, or damages awarded for pain, suffering, and loss of companionship, do not appear to affect the quality of care. The authors, however, highlight the limitations of addressing this question solely from the lens of a damages-cap analysis. Legislatures can modify tor t rules beyond just limiting the extent of the harm posed by the current system. They might attempt to adopt a new system altogether—for instance, a system that sets the standards to which physicians are held in an entirely different light. Frakes and Jena test for changes in healthcare quality when states used national standards to measure physicians’ care. In states where healthcare quality was rated more highly before using national standards, adopting national standard laws did not seem to change the quality of care. But in those states that initially had low-quality care, adopting national standards was linked to improvements in all quality measures.