Education research gets an F.
Since holding teachers responsible for student performance is now all the rage, from the White House to the political right, let us do a simple thought experiment. Imagine an amateur baseball league in which team owners dictate which bats players use. The owners try to choose the best, but the research on bats is so poor, they have to rely on anecdotes—“Barry Bonds hit 73 home runs with maple!”—and on manufacturers’ claims. As a result, some teams wind up using bats that are too heavy, too fragile, or no better than a broomstick. Does it make sense to cut players who were forced to use ineffective equipment?
It goes without saying that effective teaching has many components, from dedication to handling a classroom and understanding how individual students learn. But a major ingredient is the curriculum the school requires them to use. Yet in one of those you’ve-got-to-be-kidding situations, the scientific basis for specific curricular materials, and even for general approaches such as how science should be taught, is so flimsy as to be a national scandal. As pay-for-performance spreads, we will therefore be punishing teachers for, in some cases, using the pedagogic equivalent of foam bats. “There is a dearth of carefully crafted, quantitative studies on what works,” says William Cobern of Western Michigan University. “It’s a crazy situation.”
Cobern tried to fix that in a study comparing direct instruction with inquiry learning, competing ways to teach science. The smart money has been on the latter, in which students explore a question on their own by, say, growing some seedlings in a closet and others on a windowsill to discover photosynthesis rather than being given the concept by the teacher. Cobern’s team randomly assigned 180 eighth graders (randomization is the gold standard for research, as in trials for new drugs) to one or the other form of instruction, they report in a study published in Research in Science & Technological Education in April. Contrary to received wisdom, “as long as students are actively engaged, direct instruction does just as well as inquiry-based teaching” in how well kids learn science concepts, he told me. Yet national and state standards push inquiry learning. As Cobern’s team diplomatically put it, “Some claims for inquiry methods regarding understanding the nature of science are not sufficiently supported by evidence.”
Indeed, an exhaustive analysis of 138 studies of inquiry-based science instruction in K–12 found that most of them had highly problematic designs: 53 percent did not randomly assign students to one kind of instruction or failed to include a control group. Not only did most studies have “marginal methodological rigor,” the analysts found, but the trend was “toward a decrease” in rigor.
When it comes to specific curricula, the scientific vacuity of education research is even more exasperating. In 2002 Congress established the Institute of Education Sciences in the Department of Education to conduct peer-reviewed studies of everything from math software to rewarding kids for academic performance. If you have a child in K–12, peruse IES’s What Works Clearinghouse. It is an ugly picture. Only one of five studies of Saxon Math (a home-schooling program for grades 6 to 8) met the standards for scientific rigor, IES found. For Web-based Odyssey Math, a K–8 curriculum, no studies out of 20 did so. Of 12 studies of Singapore Math, modeled on that city’s supposedly superior math program, not a single one was methodologically rigorous (in many cases, there was no comparison group). Of 23 studies of Bridges in Mathematics, not a single one met evidence standards, and 16 were so sloppy that it was “impossible to attribute the observed effect” to the program. None of the 40 studies of Investigations in Number, Data, and Space met the standards.
And where the studies were rigorous, the curriculum often flunked. None of four LeapTrack math programs “demonstrated significant effects on student achievement.” For Plato Achieve Now, which runs on a PlayStation Portable and emphasizes learning at one’s own pace, there was “no discernible effect on math achievement.” Ready, Set, Leap!, a preschool reading curriculum, “was found to have no discernible effects on oral language, print knowledge, phonological processing, early reading/writing skills, and math,” based on two good studies. And so on.
Philosopher Alfred North Whitehead wrote that when one compares the importance of education with “the frivolous inertia with which it is treated,” it is “difficult to restrain within oneself a savage rage.” That was 80 years ago.