Matching Administrative Data to Inform Policy

New network sets goals, begins to build community

Network members

Clockwise, from far right: Jens Ludwig of the University of Chicago talks with Katherine Magnuson of the University of Wisconsin-Madison, Harvard's Raj Chetty, and IPR's Quincy Thomas Stewart after a workshop session on the long-term impacts of teachers.

The federal government has spent more than half a billion dollars so far on building longitudinal, state-level data sets around the nation. While it has become a national priority, states’ data collection efforts are still in their infancy, with little in the way of best practices or minimum guidelines to optimize data collection, use, or a host of related issues.

On October 10 and 11, more than 50 academics, policymakers, and practitioners gathered at Northwestern University for an inaugural meeting that seeks to establish an interdisciplinary network connecting these three groups around building state-level matched data sets. The new network is an initiative of the National Science Foundation.

The Network

Academics are finding that such sets are invaluable for conducting first-order empirical research to develop and evaluate education policies and practices. Such data sets are also critical for evaluating short- and long-term outcomes, especially when the data are linked across different areas such as health, education, and labor force participation. In the wake of high-stakes testing and value-added initiatives, states have become interested in collecting such longitudinal, interwoven data to better evaluate programs and policies. Yet to date, partnerships between these two groups are rare—though successful ones have been built in states such as Florida, North Carolina, and Michigan, including by the workshop co-host universities of Northwestern and Duke.

Network members
The inaugural network members

Several administrators involved in their states’ data collection efforts were also on hand to share state-level views on data collection and potential partnerships. They were Massachusetts’ Carrie Conaway, Arkansas’ Neal Gibson, and Michigan’s Joe Martineau.

The network will also benefit from an advisory board that includes several former state governors, including Bob Wise of West Virginia and Bev Purdue of North Carolina, state education superintendents like Christopher Koch of Illinois, and former policymakers and practitioners such as Grover Whitehurst of the Brookings Institution, who was founding director of the Institute of Education Sciences, and Patricia Levesque, who directs two education foundations and was Jeb Bush’s deputy chief of staff when he was governor of Florida.

“There is no national database. States are figuring how to build longitudinal data systems and providing actionable information to policymakers, parents, and teachers, and they are learning from one another,” said advisory board member Aimee Guidera, founder and executive director of Data Quality Campaign. “This is why this effort—to better connect research to the today’s critical questions of policy and practice—is so important.”

Said IPR Director David Figlio, a workshop co-leader: “The network’s main aim is to develop an infrastructure that will help to further research that responds to states’ needs. By building bridges between state-level policymakers with data sets and scholars, we hope to create a two-way flow that will eventually benefit education policy and scholarship in the U.S. and the world.” 

The Research

To demonstrate how such data can be used and what it can reveal, presentations of policy-relevant research were made by some of the nation’s leading experts in harnessing big data sets.

“The presentations offered examples of how groups of researchers and groups of state-level people have successfully collaborated,” said Duke’s Kenneth Dodge, who co-led the workshop along with Figlio.

Sunny Ladd
Helen Ladd of Duke discusses results from two
early childhood initiatives in North Carolina.

Early childhood policies
Helen Ladd of Duke University presented results from two early childhood initiatives in North Carolina on third-grade outcomes. Ladd and colleagues matched birth records to third-grade test scores over time and looked at how the receipt of state funds at the county level translated into community-wide outcomes. The results revealed moderately large positive effects of the programs on third-grade test scores, with larger effects for children whose mothers have low education. Next, Figlio shared results of a study looking at early childhood intervention for children with autism. Using matched birth and school records from Florida for more than 8,000 autistic students, he and his collaborators found that early intervention had a substantial positive effect on behavioral and cognitive outcomes for autistic students. Figlio said the results suggested significant new policies, such as standardizing a universal screening for autism during doctors’ visits before the age of 2. Sandra Black of the University of Texas at Austin shared research looking at the effects of childcare subsidies on student performance in Norway. Using Norway’s detailed population data, along with municipality-level childcare subsidy data, Black and her colleagues compared families right above and below the income cut-off for the lowest subsidy, showing it had a significant, positive effect on tenth-graders’ achievement. They also found that the benefit acted like an income transfer, increasing the wealth of the families receiving the highest subsidy by up to 10 percent.

Policies involving teachers and school leaders
Harvard’s Raj Chetty investigated whether evaluating teachers on the basis of their test score “value-added” could help improve students' earnings, college attendance rates, and other markers of success. Using data on 1 million children in a large urban school district, Chetty and his colleagues found that value-added estimates based on test scores provide "good forecasts of teacher quality.” Their research also underscored a much broader implication: Improving teacher quality is likely to have large returns, no matter the metric used.
While researchers often focus on differences in schools, Stanford’s Eric Hanushek is concerned about how a school’s quality might change over short periods of time. Analysis consistently shows large differences among teachers, with 15 to 20 percent of teachers leaving their school each year. Similarly, recent analysis of Texas schools shows large differences of value-added by principals, adding 2 to 7 months of learning in a year, along with high turnover rates. Thus, examining school quality over time requires consideration of teachers’ and principals’ high turnover rates that, in turn, can directly affect student learning and growth. His colleague Susanna Loeb spoke about her research that tracks how and what principals do on a daily basis and the effect of their actions on teacher quality and ultimately student learning. She and her team have partnered with several urban school districts, combining student, staff, course, and school-level data with surveys, observations, and interviews. After five years of data collection and analysis, they have determined that primary data is most useful when combined with administrative data. The two latter presentations hold important implications for measuring principals’ effectiveness and by extension school/teacher quality and student outcomes.

Policies regarding risky teenage behaviors
Jens Ludwig of the University of Chicago described a violence prevention program that teaches high-risk youth to “stop, look, and listen” to overcome automatic—and potentially violent—responses in high-stakes situations. Using two randomized experiments, his team was able to assess both programs’ effectiveness in Cook County—in part due to the cost effectiveness of combining administrative data sets for 2,700 youth in one and benefitting from a low-cost natural experiment for the other. The evaluations revealed potential returns of between $2 and $30 in societal benefits for each $1 spent.

David Figlio and Kenneth Dodge
IPR Director David Figlio (left) talks with fellow workshop
co-leader Kenneth Dodge
about research on improving teacher
quality presented earlier in the day.

His colleague Elaine Allensworth of the Consortium on Chicago School Research then dove into an ongoing study of school discipline policies, instruction, and student achievement. The researchers tied Chicago Public School (CPS) administrative data to survey, census, juvenile justice, police, and child and family services’ data to study changes in CPS’ recent moves towards less punitive disciplinary actions. They are examining effects for 150,000 students each year over six years. Duke’s Elizabeth Ananat recounted a project that set out to determine if teen fertility rates were affected by job losses in the same way as those of adult women. She and her team combined North Carolina labor data with birth records and found that job losses decreased fertility rates for black teens but not for whites. In addition, in the face of job losses, black teen girls were more likely to use contraception and had lower fertility rates if they were on track in school or more financially advantaged.

“I've been in this game for a while, and I was really impressed by what I saw over the past two days,” said Tim Sass of Georgia State University in speaking about the various presentations of studies using large, matched administrative data sets. “I know a condensed version of this would be very useful to others to show that this is what you can do with these new data sets that are coming online.”

Photos: Jim Ziv and Patricia Reese

This article was updated on 10/22/13.