Improving No Child Left Behind
IPR briefing provides research-driven recommendations for policymakers
From right to left: Thomas D. Cook, Diane Whitmore
Schanzenbach, and David Figlio discuss their findings.
At an IPR policy research briefing in Washington, D.C., three of the Institute's researchers spoke to a crowd of 65 Capitol Hill staffers, government officials, researchers, and advocates in February 2010. They discussed what lawmakers should consider when they retool No Child Left Behind (NCLB).
- NCLB: A Brief Overview
- Is It Working?
- Left Behind by Design
- Aims, Games, and Accountability
- Where to Next?
The first question is simply: Has NCLB raised achievement? IPR social psychologist and education professor Thomas D. Cook presented one of the most scientifically rigorous sources of evidence to date to show that NCLB has indeed raised standardized test scores in public schools since 2002.
Cook, with IPR researchers Manyee Wong, the paper’s lead author, and Peter Steiner, compared public school students nationally to students in Catholic and non-Catholic private schools, which are not subject to NCLB. Using data from the National Assessment of Educational Progress (NAEP), Cook demonstrated that public school students around the nation made gains in fourth- and eighth-grade math but not in reading.
Thomas D. Cook
For the nation's public school fourth-graders, NCLB raised math achievement by an average of six months (standard deviation of 0.28) between 2002 and 2009, “narrowing the gap between public and private schools by about half,” said Cook, who is Joan and Sarepta Harrison Chair in Ethics and Justice at Northwestern.
“These are very large gains,” Cook emphasized, later noting his surprise at how quickly they showed up once the law was implemented.
Given that each state is free to use either an easy or hard achievement test and to designate its own standards for declaring students proficient, Cook, Wong, and Steiner then drilled into a state-level comparison. Looking at NAEP scores before and after NCLB was implemented, they find that a state’s improvement in math is significantly tied to how difficult it is to attain a state's proficiency threshold since an easy threshold means that few schools in a state will need to reform under the law while a more stringent one means that many will have to.
For example, 35 percent of students reached proficiency on the NAEP, on average, across public schools in both Maine and California. In Maine, the average passing rate on the state test linked to NCLB was about the same—37 percent. But in California’s schools, where the proficiency threshold was much lower, 84 percent of students passed the state test. As a result, many more schools in Maine than in California failed to make adequate yearly progress (AYP) and faced sanctions or forced reforms—sometimes as serious as firing all of the teachers or shutting down completely.
Bringing these methods to bear across all 50 states, the researchers found that states with lower thresholds for proficiency did not gain as much in math by 2009 as those states with higher thresholds.
“The bottom line, particularly for math, is that higher standards matter,” Cook said. Setting a national threshold for making AYP would also cut down on states’ ability to game the system and help raise achievement, he added.
If the original goals of NCLB were to “leave no child behind” and close the black-white, rich-poor, and Hispanic-white achievement gaps, NCLB has not lived up to its promises, recounted Diane Whitmore Schanzenbach, currently an IPR economist who was with the University of Chicago at the time of the briefing.
Diane Whitmore Schanzenbach
“In Chicago, we commonly see that kids at the bottom are left behind—and in some cases, kids at the top as well,” she said.
Schanzenbach and her colleague Derek Neal took advantage of the unique availability of longitudinal test score information on individual students in Chicago Public Schools (CPS) to study how the 2002 implementation of NCLB affected students. The researchers followed two groups of children: a pre-reform group that was given “low-stakes” tests in third grade in 2000 and then again in fifth grade in 2002, and a post-NCLB group that was given a low-stakes test in third grade in 2001 and then a high-stakes test used to define their schools’ NCLB passing status in fifth grade in 2003.
They then divided each group into deciles based on their reading and math scores. In both subjects, they found scores improved for children in the third through ninth decile range under high-stakes testing relative to low-stakes testing, while scores remained the same for the highest- and lowest-scoring students—even dropping in math for the lowest 10 percent.
Schanzenbach’s findings were consistent with other qualitative research findings that document evidence of teachers and schools providing extra attention to those children clustered around the proficiency threshold. She likened the practice to “educational triage,” where those with the best chances of passing receive treatment, while the rest are left behind.
“You might think, ‘Oh, one or two deciles at the bottom might not be that big of a deal,’ but if we are looking across all grades in Chicago, this is 25,000 to 50,000 kids,” Schanzenbach said.
“So as we seek to reauthorize the law, can we improve some of these misaligned incentives?” she asked. “I think the answer is ‘yes.’”
Under the current law, only passing or failing the test matters, Schanzenbach emphasized.
“There is no credit at all for moving a student’s test score from a very low score to almost passing,” she explained.
As a result, she continued, there is no incentive for schools to concentrate on low-achieving students who are unlikely to improve enough to pass the test. Furthermore, the Obama administration’s desire to raise NCLB passing standards implies that there will be even more low-achieving children who have little chance of passing the test.
Thus, Schanzenbach would like to see lawmakers implement a new system, similar to the one in Massachusetts, that gives partial credit for moving children up the achievement ladder, even if they do not yet pass the test.
IPR education economist David Figlio also seconded the evidence that high standards have led to improved test performance but pointed to some of NCLB’s unintended—and unfortunate—consequences.
Over the past five years, he has investigated the wide-ranging effects of several high-stakes accountability systems, including NCLB, examining such topics as teacher behavior and retention, student diets, real-estate markets, and donations to schools.
“There are strong incentives for schools to engage in a lot of different ways of trying to make themselves look better,” Figlio remarked.
Especially in the ongoing quest for ever-higher test scores, Figlio has uncovered several ways in which schools try to “game the system” and avoid serious reforms. In one study of Virginia public schools, he and colleague Joshua Winicki found that several districts upped the calories in their menus on testing days in an attempt to “juice” the scores—but only at the schools faced with potential sanctions under the state’s accountability system.
Research has shown that a system based on gains removes such incentives for a one-time score boost and refocuses schools on learning. Figlio and his colleagues Cecilia Rouse, Jane Hannaway, and Dan Goldhaber found promising results in a study of public schools in Florida, which replaced its single threshold approach with a value-added one in 2002. Under the new system, schools paid more attention to low-performing students and spent more time on high-stakes subjects. They also increased teacher resources and implemented policies to improve the performance of the worst ones.
In addition to reducing gaming, this gains-based system seemed to benefit students across the board, counteracting the learning “bubble” that Schanzenbach described. But a gains-based system could have its own adverse effects, Figlio warned.
For one, it might remove some of the focus from traditionally disadvantaged groups. In addition, it might upset communities where students are mostly proficient already, since it would be practically impossible—and unnecessary—for their schools to produce sweeping gains from year to year.
Figlio presented evidence from his study of Florida housing markets to emphasize the point that school performance matters not just to schools, parents, and teachers but also the wider community. He and colleague Maurice Lucas traced fluctuations in local housing markets in Florida to the recently implemented school report card system. They found consistently higher housing prices in areas with “A”-rated schools, while housing prices dropped where schools were deemed to be failing.
Given these competing conditions, what is Figlio’s solution? A hybrid system for accountability.
Building on Cook’s evidence, it seems clear, Figlio said, that schools should set a high-proficiency threshold, but also measure gains for those students who do not achieve it. They should provide additional means for those students far away from the threshold or students in targeted groups to avoid the bubble effects described by Schanzenbach.
As for now, gaming the system works, Figlio said. “So we need to be careful about what we expect out of these test scores.”
While all three experts agreed that achievement gains have been realized under the current law, it clearly needs to be improved. The evidence gathered from their research points to an accountability system that is built on a common set of high standards and that takes into account student improvement across the achievement spectrum, either by measuring gains or implementing multiple thresholds. It would also recognize that what happens in schools has wider effects beyond just educating the nation’s children and would acknowledge those high-performing schools where achievement might have reached a plateau.
Finally, for all the talk about accountability and measurement, Cook—who was part of the committee to review Title I, the funding mechanism for NCLB—strongly encouraged policymakers to step back and take a broader look at the issue.
“I believe in a ‘whole-child’ view,” Cook said, which he noted is unfortunately often overlooked in the ongoing debate over education reform.
NCLB was signed into law in 2002 and has been one of the most far-reaching overhauls of U.S. education policy to date. Due to be reauthorized, NCLB requires states that receive federal education funding to set educational standards and administer annual reading and math tests to students from third to eighth grade. Each state is free to select its own tests and set its own proficiency standards, but schools are expected to make adequately yearly progress (AYP) in helping all students reach proficiency in reading and math by 2014. While NCLB was hailed by some as a much-needed step in the direction of standards-based accountability, critics of the law accuse it of encouraging teaching to the test; unfairly penalizing minority, gifted, low-income, and special-needs students; and using overly punitive sanctions for under-performing schools, such as replacing teachers or closing schools entirely.
Back to the top.