Recent Performance Measurement and Rewards Research

Performance in Government & Nonprofits

Patents and Scientific Advances
Scientific research can lead both to basic insights into the world and to practical applications. It has been hard to measure, however, the extent to which scientific advances have led to technological innovations. How much does the research from universities and government labs contribute to new inventions with practical applications that firms can manufacture? In research published in Science, strategy professor and IPR associate Benjamin Jones, along with his Kellogg colleague Mohammad Ahmadpoor, develop a way to measure the relationship between scientific research and patents. The researchers connected references from all 4.8 million U.S. patents from 1976–2015 to all 32 million journal articles published from 1945–2013, allowing them to determine the distance between an article and a patent. Beyond whether published scientific research connected to patents, they also measured how many steps—other articles or patents—were connected along the way. Jones and Ahmadpoor discover that 61 percent of patents link back to a prior research article, and 80 percent of cited research articles link forward to a future patent. They are able to characterize the connections by scientific field, types of institutions, for example, government labs, and even individual scientists. Jones is the Gordon and Llura Gund Family Professor of Entrepreneurship.

Benjamin Jones
Strategy professor and IPR associate Benjamin Jones has
developed a way to measure the relationship between
scientific research and patents, discovering that 61
percent of patents link back to a prior research article.

Compulsory Licensing and Innovation
Compulsory licensing allows governments to license patented inventions without the patent owners’ consent. Developing countries often use this policy to license domestic firms to manufacture valuable and otherwise expensive drugs, such as those to treat HIV and cancer, that are patented in foreign countries. The patent holders do not give permission for the license and do not receive payment for the use of their invention. Does compulsory licensing discourage discoveries that lead to patents by reducing their value? Or, does it spur innovation by motivating inventors to stay in front with new breakthroughs? To answer this question, economist and IPR associate Nicola Bianchi and his colleagues examine a rare historical example of compulsory licensing in the Journal of Development Economics. During World War I, the U.S. government confiscated all U.S. patents by German inventors, making more than 4,700 of them available for compulsory licensing. In the fields most affected, German inventors began to apply for more patents after the war, the researchers find. Compared with the number of pre-war patents in chemicals, for example, patenting increased 30 percent after 1918. The data suggest that compulsory licensing might have encouraged inventors by increasing competition, motivating them, thus, to invest more in research and development.

Cash for Carbon
Deforestation accounts for an estimated 9 percent of human-induced carbon emissions. How can deforestation be slowed effectively in developing countries, where most global deforestation occurs? A recent study in Uganda by IPR economist Seema Jayachandran and her colleagues, published in Science, suggests that paying people to conserve their trees could be a highly cost-effective way to reduce deforestation, and, therefore, carbon emissions. The researchers conducted a randomized controlled trial in 121 villages in western Uganda over two years. In 60 villages, forest owners were paid to not cut down their trees, while forest owners in the remaining villages received no payments. Using high-resolution satellite imagery to compare tree cover, Jayachandran and her colleagues determine that forest cover declined by just 4.2 percent in villages that received payments. For villages not enrolled in the payment program, forest cover fell by 9.1 percent, meaning fewer trees were cut down when payments were offered. Not only did the program effectively reduce deforestation, it was very cost-effective. The researchers estimate that the benefit of delayed carbon dioxide emissions is 2.4 times larger than the program costs. The Global Environment Facility, through the United Nations Environment Program, International Initiative for Impact Evaluation, and National Science Foundation provided project funding (see p. 88).

Lessons from Two Years at Hamilton
Diane Whitmore Schanzenbach, IPR director and economist, gave an overview of The Hamilton Project under her leadership from 2015–17, paying particular attention to measuring its performance, in the IPR Performance Measurement and Rewards Series on December 5. She identified three types of Hamilton products and the indicators of their success. The first, Hamilton’s “bread and butter,” consists of identifying policy proposals from leading academics and making them understandable, actionable, and available. The second product is creating a primer to outline the facts of an issue, and the third is producing a brief but timely economic analysis of an issue. What were the markers of success for these efforts? Both the number and the quality of the publications and events are important in evaluating performance. Measures include good press coverage, both short- and longer-term; citations by federal and state policymakers; the size of the audience at the release and its composition; and the number of posts, tweets, and downloads. Schanzenbach, the Margaret Walker Alexander Professor, pointed to one of Hamilton’s “biggest successes” under her tenure: its report on states’ options for a new metric required by the Every Student Succeeds Act (ESSA). It recommended states adopt chronic absenteeism as that metric, and 36 of 50 states did so, as well as Puerto Rico and Washington, D.C. However, she reminded her audience, much assessment of policy impact is not measurable.

The Perils of Pay-for-Performance
Government and nonprofit activities are often difficult, if not impossible, to measure and assign monetary value to in ways that are broadly acceptable. These difficulties prevent the adoption of incentive systems that align rewards with performance. IPR economist Burton Weisbrod is writing a book, under contract with Stanford University Press, to consider the unintended—but foreseeable and counterproductive—consequences of the rising tide of efforts to reward performance measures, such as standardized tests for schoolchildren and crime rates for police. These involve adopting measures that are both incomplete and biased and then rewarding them. Titled “The Perils of Pay-for-Performance: Why Strong Rewards in Government and Nonprofit Organizations Don’t Work,” the book will cover a wide array of public and nonprofit sector services, such as K–12 education, hospitals, nursing homes, hospices, policing and jails, museums, charities, and the federal judiciary. Weisbrod emphasizes how the forces at work in these services cause strong rewards to be strategically gamed, inflating or distorting results, so that measured performance, not truly good performance, is rewarded. He explains why measured performance and actual performance are in conflict—why, for example, larger rewards in public schools might improve test scores but not learning. He notes that the next frontier for pay-for-performance is higher education and predicts the forms gaming will take there. Weisbrod directs IPR’s Performance Measurement and Rewards Series, which features presentations by researchers studying healthcare, education, policing, and other social service industries. He is Cardiss Collins Professor of Economics.

Increasing Enrollment in SNAP
The Supplemental Nutrition Assistance Program (SNAP), previously known as food stamps, provides cash that can be spent on food to eligible low-income U.S. households. The program requires individuals to apply and demonstrate their eligibility to qualify. Some apparently eligible people do not apply for benefits, perhaps put off by the time and effort required. How can they be encouraged to apply? IPR economist Matthew Notowidigdo and Amy Finkelstein of the Massachusetts Institute of Technology are examining several possibilities in research sponsored by the Abdul Latif Jameel Poverty Action Lab. With research partner Benefits Data Trust (BDT), a nonprofit experienced in helping people obtain benefits, Notowidigdo and Finkelstein conducted a randomized controlled trial among elderly households in Pennsylvania that were likely eligible. About 30,000 such households were divided into three groups, all of whom received a letter about their possible SNAP eligibility: one group also received a follow-up postcard; a second group was given a toll-free number for BDT’s application assistance; and a control group received no assistance from BDT. The researchers measured the impact of the two methods—information only and information plus assistance—by the number of households who applied and ultimately enrolled in SNAP after nine months. The group that received information only increased enrollment by 5 percentage points, while the group that received information plus assistance increased enrollment by 12 percentage points. The researchers are developing our understanding of how to improve outreach methods to low-income households who typically do not participate in social safety net programs.

Improving Earthquake Hazard Maps
Scientists create complex models to forecast the hazards that earthquakes pose, both to structures and to people, and they represent their forecasts in maps that show different levels of hazards for different geographical areas. IPR statistician Bruce Spencer, geophysicist and IPR associate Seth Stein, and IPR graduate research assistant Edward Brooks are researching ways to improve the accuracy of hazard maps so that better policy decisions can be made about where to locate vulnerable structures—such as nuclear power or chemical plants—and for setting appropriate building codes. In an International Journal of Earthquake and Impact Engineering article, the researchers explore how hazard maps actually perform. Using a 510-year record of shaking in Japan, they find that too much historical detail makes the maps less accurate for predicting future hazards. They conclude that the goal of such hazard maps should not be to build the most detailed model, but instead, one that is robust or stable in the sense that the forecast is not unduly affected when the earth does not behave exactly as expected. Stein is William Deering Professor.

Accountability in Education

School Policies Unevenly Implemented
In 2002, the Florida legislature implemented rules that require students who do not meet a test of grade-level reading skills to repeat third grade. The policy includes certain exemptions for students that allow them to continue to fourth grade. IPR education economist David Figlio, the Orrington Lunt Professor, with former IPR graduate research assistant Christina LiCalsi and Umut Özek, who are both at the American Institutes for Research, investigate whether the retention rule was applied to all children whose test scores fell below the cutoff, or if some children received exemptions based on their socioeconomic (SES) background. A good indicator of higher SES—maternal education level—is recorded in the state’s birth records data, which the researchers matched to school records. The researchers determine that the grade retention policy is, in fact, not enforced equally. Among the students who scored just below the reading-level cutoff, those whose mothers had not graduated from high school were 14 percent more likely to be retained than children whose mothers had a bachelor’s degree or more. The ability of more highly educated mothers to know how to apply for exemptions and how to deal with school personnel most likely leads to the discrepancy in applying the retention rule, the researchers suggest. Since evidence shows that grade retention of lower-SES children might be harmful, it could be that Florida’s test-based promotion policy might be harming, not helping, these students. The study was published in Education Finance and Policy and received funding from the U.S. Department of Education’s Institute of Education Sciences.

Healthcare Markets and Regulation

Joe Feinglass
Feinberg researcher and IPR associate 
Joe Feinglass explains how emergency
department visits in Illinois have changed
since passage of the Affordable Care Act.

The ACA and Emergency Department Use
Of the 1.5 million people in Illinois who were uninsured prior to the Affordable Care Act (ACA) becoming law in 2014, 650,000 now have some sort of insurance, mostly Medicaid. Research professor of medicine and IPR associate Joe Feinglass has been tracking newly insured people’s increased emergency department (ED) visits in the state since that time. In research using statewide hospital data from 2011–15, published in the Annals of Emergency Medicine, Feinglass and his colleagues discover that adults aged 18–64 made more ED visits after the ACA’s expansion. Medicaid beneficiaries made more visits and uninsured patients made fewer visits. In the Western Journal of Emergency Medicine, Feinglass and fellow researchers compare people’s insurance status change and ED visit rate for two years prior and two years after the ACA’s enactment. They confirm that its implementation led to an increase in ED visits by adults aged 18–64, while areas with the largest increases in Medicaid enrollment experienced the largest growth in ED use. Will providing better access to primary and preventive care to newly insured adults reduce ED use over time? While more research is needed to answer this question, the researchers stress that monitoring local ED use is vital to the success of new home- and community-based care coordination initiatives.

What Does Health Insurance Do?
Publicly subsidized health insurance coverage, such as that provided through the ACA, provides benefits to the previously uninsured, including improvements in health, reductions in out-of-pocket spending, and reduced medical debt. Newly insured people are not the only beneficiaries, however, of expanding formal health insurance coverage, as uncovered by Notowidigdo, Finkelstein, and Neale Mahoney of the University of Chicago. They demonstrate that because the nominally uninsured pay only a small share of their medical expenses—about one-fifth to one-third—health insurance also provides substantial payments to others, such as hospitals and healthcare providers, who would otherwise have to cover the costs of providing uncompensated care to the uninsured. In effect, the uninsured are implicitly insured. This helps explain why individuals do not enroll in heavily subsidized public health insurance and why they value formal health insurance at costs substantially less than the cost to insurers of providing them coverage. Expanded, publicly funded health insurance might not be benefiting low-income people as much as it might be transferring income to healthcare providers and others who care for the uninsured. The National Institute on Aging supported their research.

Economic Results of Hospitalization
Fewer Americans are uninsured since the ACA’s passage. But even insured patients might face a big financial price if they are admitted to the hospital. To assess the economic impacts of hospital admissions, Notowidigdo, along with colleagues from the University of California, Santa Cruz, and the Massachusetts Institute of Technology, closely examine the health and financial records of Californians who entered the hospital between 2003 and 2007 in the American Economic Review. Excluding pregnancies and people who had been in the hospital in the last three years, the researchers find the average hospital stay lasted about four days and had a sticker price of $45,000—which is typically bargained down in later negotiation. Uninsured patients quickly fell into debt, while working-age insured adults also suffered financial difficulties. Following a hospital stay, these workers with health insurance experienced larger out-of-pocket medical spending, unpaid medical bills, and bankruptcy, as well as reduced earnings, access to credit, and consumer borrowing. For example, insured adults between the ages of 50 and 59 saw their annual incomes fall by over $9,000, a decline of about 20 percent, and their probability of being employed dropped by 15 percent after hospitalization. In contrast, elderly patients, who had health insurance through Medicare and had small labor market earnings, appeared to suffer few or no economic setbacks from being hospitalized. These findings underline that insurance in the United States, unlike some European countries, rarely covers any decline in earnings. The National Institute on Aging and the NSF provided research funding.

Nonprofit Hospitals Respond to Change
In research published in the RAND Journal of Economics, healthcare economists and IPR associates David Dranove and Craig Garthwaite, along with Kellogg’s Christopher Ody, reveal mixed evidence about how much nonprofits behave like for-profits. They discover that the average nonprofit hospital suffered reduced assets from the 2008 collapse, which economic theory suggests should lead to them raising prices, reducing quality, or taking other actions to decrease their unprofitable activities. Yet they did not raise prices, cut operating costs, or decrease their offering of charity care—unlike how a for-profit would act. Their financial losses thus persisted for several years. The nonprofits did cut back some relatively unprofitable services, however. The researchers show that nonprofits with bigger market shares adjusted prices without changing the services they offered, a response the researchers did not expect from for-profit hospitals. For the most part, however, nonprofit hospitals behaved like for-profits in disguise, responding to wealth shocks in the same way that for-profits would respond. Dranove is the Walter McNerney Distinguished Professor of Health Industry Management, and Garthwaite is the Herman Smith Research Professor in Hospital and Health Services Management.