Content uploaded by Richard P Phelps
Author content
All content in this area was uploaded by Richard P Phelps on May 17, 2015
Content may be subject to copyright.
A preview of the PDF is not available
Dismissive Reviews: Academe’s Memory Hole
Abstract
In scholarly terms, a review of the literature or literature review is a summation of the previous research that has been done on a particular topic. With a dismissive literature review, a researcher assures the public that no one has yet studied a topic or that very little has been done on it. A firstness claim is a particular type of dismissive review in which a researcher insists that he is the first to study a topic. Of course, firstness claims and dismissive reviews can be accurate—for example, with genuinely new scientific discoveries or technical inventions. But that does not explain their prevalence in nonscientific, nontechnical fields, such as education, economics, and public policy, nor does it explain their sheer abundance across all fields.
... But, they may also have been enticed by professional rewards. Once the think tank elite publicly supported some of the establishment doctrine on assessment, 8 they were 7 Those declaring the large literature on testing's effects to be nonexistent include Greg Cizek, David Figlio, Jay P. Greene, Erik Hanushek, Rick Hess, Brian Jacob, Daniel Koretz, Helen Ladd, Tom Loveless, Maurice Lucas, Margaret Raymond, Sean Reardon, and Melissa Roderick (see Phelps, 2012a). 8 Some elements of establishment assessment beliefs are: (1) there is no, or almost no, research finding any benefits to high-stakes testing; (2) standardized educational testing, particularly when it has stakes, is enormously costly in monetary terms; (3) there exists substantial evidence that high-stakes tests cost plenty in nonmonetary terms, too-they "distort" instruction, narrow the curriculum, etc.; (4) all high-stakes testing is prone to "test-score inflation"-artificial rises in average test scores over time due to "teaching to the test"; (5) no-or low-stakes tests, by contrast, are not susceptible to test-score inflation because there are no incentives to manipulate scores; (6) as score trends for high-stakes tests are unreliable and those for no-or low-stakes tests are reliable, no-or low-stakes tests may be used validly as shadow tests to audit the reliability of high-stakes tests' score invited to join high-profile national committees, panels, and commissions on assessment-even though they knew little about assessment-which helped them bulk up their CVs with impressive-sounding credentials, and paid honoraria. ...
... The establishment doesn't need to shun and ostracize most of those who publish or conduct research they do not like; the think tank elite does it for them. The think tank elite is the cork in the bottle that keeps the American public misinformed (Phelps, 2012a). ...
The education establishment doesn't need to censor and suppress most research pertinent to education reform; the think tank elite does it for them.
... Contrary research, evidence, or points of view are not mentioned or, in the most insidious cases, are openly declared not to exist. (See, for example, Phelps, 2012b.) There is no purpose to arranging a debate if the other side apparently does not exist. ...
... (See, e.g., Phelps, 2012cPhelps, , 2013.) Even dismissive reviews-through which researchers blatantly declare that contrary research and evidence do not exist-are quite common, and commonly accepted as fair play (Phelps, 2012b). ...
It requires long seasoning in the bog for one to develop a taste for which education information is accurate, which is myth, and which is just plain dishonest.
... Meanwhile, a cornucopia of studies contradicting the two research center studies have been repeatedly declared nonexistent by the same researchers and thousands of sympathetic others inside US education schools (Phelps, 2005(Phelps, , 2008(Phelps, , 2009b(Phelps, , 2012a(Phelps, , 2012b. ...
This article explains the various meanings and ambiguities of the phrase “teaching to the test” (TttT), describes its history and use as a pejorative, and outlines the policy implications of the popular, but fallacious, belief that “high stakes” testing induces TttT which, in turn, produces “test score inflation” or artificial test score gains. The history starts with the infamous “Lake Wobegon Effect” test score scandal in the US in the 1980s. John J. Cannell, a medical doctor, discovered that all US states administering national norm-referenced tests claimed their students’ average scores exceeded the national average, a mathematical impossibility. Cannell blamed educator cheating and lax security for the test score inflation, but education insiders managed to convince many that high stakes was the cause, despite the fact that Cannell’s tests had no stakes. Elevating the high stakes causes TttT, which causes test score inflation fallacy to dogma has served to divert attention from the endemic lax security with “internally administered” tests that should have encouraged policy makers to require more external controls in test administrations. The fallacy is partly responsible for promoting the ruinous practice of test preparation drilling on test format and administering practice tests as a substitute for genuine subject matter preparation. Finally, promoters of the fallacy have encouraged the practice of “auditing” allegedly untrustworthy high-stakes test score trends with score trends from allegedly trustworthy low-stakes tests, despite an abundance of evidence that low-stakes test scores are far less reliable, largely due to student disinterest.Keywords: Test security, Educator cheating, Test score inflation, High stakes, Standardized tests, Education, CRESST, Daniel Koretz, John J. Cannell, Lake Wobegon Effect. La Familia de Falacias "Enseñando para el Examen"Este artículo explica los diversos significados y ambigüedades de la frase "enseñar para el examen" (TttT: teaching to the test en inglés), describe su historia y su uso como un peyorativo, y describe las implicaciones políticas de la creencia popular, pero falaz, que las pruebas de a “gran escala” inducen TttT que, a su vez, produce una "inflación en la calificación obtenida en el examen" o ganancias em cuanto a los puntos obtenidos en la prueba. La historia comienza con el infame escándalo de la puntuación de la prueba "Lake Wobegon Effect" en los Estados Unidos en los años ochenta. John J. Cannell, un médico, descubrió que todos los estados de los Estados Unidos que administraban pruebas nacionales con referencias normativas afirmaban que los puntajes promedio de sus estudiantes excedían el promedio nacional, una imposibilidad matemática. Cannell atribuyó a los educadores el engaño y la seguridad laxa por la inflación de la puntuación de los exámenes, pero los expertos en educación lograron convencer a muchos de que las pruebas a gran escala eran la causa, a pesar de que las pruebas de Cannell no tenían ninguna fiabilidad. Exagerar las pruebas a gran escala hace que TttT hace que la falla de la inflación de la puntuación de la prueba al dogma haya servido para desviar la atención de la seguridad laxa endémica con pruebas "internamente administradas" que deberían haber alentado a los responsables políticos a exigir más controles externos en las administraciones de las pruebas. La falacia es en parte responsable de promover la práctica ruinosa en la preparación de las pruebas en el formato de prueba y la administración de pruebas prácticas como un sustituto de la preparación de la materia original. Por último, los promotores de la falacia han fomentado la práctica de "auditar" tendencias de determinadas puntuación en las pruebas a gran escala con las tendencias de puntuación presuntamente confiables de las pruebas de baja exigencia, a pesar de la abundancia de pruebas donde las puntuaciones de las pruebas a menor escala son mucho menos confiables debido al desinterés de los estudiantes. Palabras clave: Prueba de seguridad, Engaño de educador, inflación de la puntuación del examen, Pruebas a gran escala, Pruebas estandarizadas, Educación, CRESST, Daniel Koretz, John J. Cannell, Efecto Lake Wobegon.
... Meanwhile, a cornucopia of studies contradicting the two research center studies have been repeatedly declared nonexistent by the same researchers and thousands of sympathetic others inside education schools (Phelps 2003(Phelps , 2005/20092012a;2012b). ...
Elevating teaching-to-the-test to dogma, from the beginning with the distortion of Dr. Cannell’s findings, has served to divert attention from scandals that should have threatened US educators’ almost complete control of their own evaluation.[10] Had the scandal Dr. Cannell uncovered been portrayed honestly to the public—educators cheat on tests administered internally with lax security—the obvious solution would have been to externally manage all assessments (Oliphant, 2011).
Recent test cheating scandals in Atlanta, Washington, DC, and elsewhere once again drew attention to a serious problem. But, instead of blaming lax security and internally managed test administration, most educators blamed the stakes and alleged undue pressure that ensues (Phelps 2011a). Their recommendation, as usual: drop the stakes and reduce the amount of testing. Never mind the ironies: they want oversight lifted so they may operate with none, and they admit that they cannot be trusted to administer tests to our children properly, but we should trust them to educate our children properly if we leave them alone.
Perhaps the most profound factoids revealed by the more recent scandals were, first, that the cheating had continued for ten years in Atlanta before any responsible person attempted to stop it and, even then, it required authorities outside the education industry to report the situation honestly. Second, in both Atlanta and Washington, DC, education industry test security consultants repeatedly declared the systems free of wrongdoing (Phelps 2011b).
Meanwhile, thirty years after J. J. Cannell first showed us how lax security leads to corrupted test scores, regardless the stakes, test security remains cavalierly loose. We have teachers administering state tests in their own classrooms to their own students, principals distributing and collecting test forms in their own schools. Security may be high outside the schoolhouse door, but inside, too much is left to chance. And, as it turns out, educators are as human as the rest of us; some of them cheat and not all of them manage to keep test materials secure, even when they aren’t intentionally cheating.
- See more at: http://nonpartisaneducation.org/Review/Essays/v12n1.htm
... I first became interested in Hoxby's work after noticing that several reports published by NBER on education topics claimed to be the first ever to study a topic or declared that no prior research on a topic existed (Phelps, 2012a). Normally, that might not seem interesting, but in each case many previous studies had been conducted. ...
The tragic results illustrate how federal and foundation money can concentrate power to achieve exactly the opposite result from that intended. Once these small, cohesive groups captured the larger organizations, they focused their efforts on restricting entry into policy arenas to those their own circles. The careers of those inside these groups have soared. Meanwhile, the amount of objective information available to policymakers and the public—our collective working memory—has shrunk.
The stated mandates of these organizations are to objectively review all the research available; instead they promote their own and declare most of the rest nonexistent. They are mandated to serve the public interest; instead they serve their own.
Currently, too few people have too much influence over those who control the education research purse strings. And, those who control the purse strings have too much influence over policy decisions. Until folk at the Bill and Melinda Gates Foundation and the US Education Department—to mention just a couple of consistent funders of education policy debacles—broaden their networks, expand their reading lists, and open their minds to more intellectual diversity, they will continue to produce education policy failure.
It would help if they would fund a wider pool of education researchers, evidence, and information. In recent years, they have, instead, encouraged the converse—funding a saturating dissemination of a narrow pool of information—thereby contributing to US education policy’s number 1 problem: pervasive misinformation.
- See more at: http://nonpartisaneducation.org/Review/Essays/v10n1.htm#sthash.7UGgn33Q.dpuf
... • Ignoring some, most, or almost all of the relevant research and evidence while suggesting that they have surveyed the entirety of the relevant research literature (i.e., selective referencing) (Phelps, 2007); • Declaring that the research and evidence they ignore does not, in fact, exist (i.e., dismissive reviewing) (Phelps, 2012a); • Claiming that one's research work is the first, or the best, or the most thorough, or the most up-to-date, or somehow summarily better than other scholars' work, thus encouraging readers to ignore other scholars' work (and pay more attention to one's own) (Phelps, 2009); and • Diminishing other scholars' research by misrepresenting it, thereby encouraging readers to ignore that research (and pay more attention to one's own) (Phelps, 2012c). ...
Synergies itself, the “final synthesis” of the REAFISO project, runs 670 pages. The country reports accumulate another 1,500 pages or so. The ten background papers average about 50 pages each. Press some more tree pulp to accommodate the requisite press releases, talking points, or the multitude of each country’s own background papers, and, all told, REAFISO’s work took a few years, substantial commitments of resources from 26 countries, and stimulated the printing of several thousand pages.
This hefty mass represents an enormous expenditure of time, money, and effort to, essentially, get it all wrong.
With the REAFISO project, the OECD has taken sides, but appears to have done so in a cowardly manner. REAFISO staff have not described evidence and sources on multiple sides of topics, weighed them in the balance, and then justified their preference. Rather, on each controversial topic they broach, they present only one side of the story. On some topics, huge research literatures several hundred studies large are completely ignored.
It is now clear that the original promise to anchor K-12 education to higher education and backmap the Common Core Mathematics Standards (CCMS) from the upper grades down to the primary grades was empty rhetoric. Higher education has scarcely been involved at all, with the exception of the institutions that agreed to place high school students who pass a Common Core-based high school examination directly into credit-bearing freshman coursework (without remediation) in return for their states receiving “Race to the Top” grant funds.
Because the CCMS are standards for all public school students in this country, regardless of achievement level, they are low standards, topping out at about the level of a weak Algebra II course. And because this level is to determine “college readiness” as they define it (which is not remotely what our public four year college and universities currently assume it to be), it is apt to mean fewer high school students taking advanced mathematics
and science coursework before they go to college, more college freshmen with even less knowledge of mathematics than currently, and more college credit-bearing courses set at an international level of seventh or eighth grade.
However, the greatest harm to higher education may accrue from the alignment of the SAT to Common Core’s high school standards, converting the SAT from an adaptable test predictive of college work to an inflexible retrospective test aligned to and locking in a low level of mathematics. This means that future SAT scores
will be less informative to college admission counselors than they now are, and that the SAT will lose its role in locating students with high STEM potential in high schools with weak mathematics and science instruction.
Each year, thousands graduate high school academically underprepared for college. Many must take remedial or developmental postsecondary coursework, and there is a growing debate about the effectiveness of such programs. This paper examines the effects of remediation using a unique data set of over 28,000 students. To account for selection biases, the paper implements an instrumental variables strategy based on variation in placement policies and the importance of proximity in college choice. The results suggest that students in remediation are more likely to persist in college in comparison to students with similar backgrounds who were not required to take the courses.
In recent years there have been increasing efforts to use accountability systems based on large-scale tests of students as a mechanism for improving student achievement. The federal No Child Left Behind Act (NCLB) is a prominent example of such an effort, but it is only the continuation of a steady trend toward greater test-based accountability in education that has been going on for decades. Over time, such accountability systems included ever-stronger incentives to motivate school administrators, teachers, and students to perform better. Incentives and Test-Based Accountability in Education reviews and synthesizes relevant research from economics, psychology, education, and related fields about how incentives work in educational accountability systems. The book helps identify circumstances in which test-based incentives may have a positive or a negative impact on student learning and offers recommendations for how to improve current test-based accountability policies. The most important directions for further research are also highlighted. For the first time, research and theory on incentives from the fields of economics, psychology, and educational measurement have all been pulled together and synthesized. Incentives and Test-Based Accountability in Education will inform people about the motivation of educators and students and inform policy discussions about NCLB and state accountability systems. Education researchers, K-12 school administrators and teachers, as well as graduate students studying education policy and educational measurement will use this book to learn more about the motivation of educators and students. Education policy makers at all levels of government will rely on this book to inform policy discussions about NCLB and state accountability systems. © 2011 by the National Academy of Sciences. All rights reserved.
This article analyzes the impact of high-stakes testing in Chicago on student achievement in grades targeted for promotional decisions. Using a three-level Hierarchical Linear Model, we estimate achievement value added in gate grades (test-score increases over and above that predicted from a student’s prior growth trajectory) for successive cohorts of students and derive policy effects by comparing value added pre- and postpolicy. Test scores in these grades increased substantially following the introduction of high-stakes testing. The effects are larger in the 6th and 8th grades and smaller in the 3rd grade in reading. Effects are also larger in previously low-achieving schools. In reading, students with low skills experienced the largest improvement in learning gains in the year prior to testing, while students with skills closer to their grade level experienced the greatest benefits in mathematics.
This article explores the basis of negative sentiments toward and current critiques of high-stakes student testing from within the education profession. To promote some balance for current policy debates, evidence for 10 unintended, unrecognized, or unarticulated positive consequences is provided. The article concludes with an examination of the relationship between high-stakes testing and accountability systems.
The impact of high school graduation exams on student achievement and dropout rates is examined. Using data from the National Educational Longitudinal Survey (NELS), this analysis is able to control for prior student achievement and a variety of other student, school, and state characteristics. It was found that graduation tests have no significant impact on 12th-grade math or reading achievement. These results are robust with a variety of specification checks. Although graduation tests have no appreciable effect on the probability of dropping out for the average student, they increase the probability of dropping out among the lowest ability students. These results suggest that policymakers would be well advised to rethink current graduation test policies.
This article briefly reviews the current discussion of the effects of test administration conditions (i.e., testing stakes), and the motivational levels associated with them, on achievement test performance. The non-experimental study presented here investigates whether differences in test administration conditions and presumed levels of motivation engendered by different testing environments affect student performance on National Assessment of Educational Progress (NAEP) administrations. The testing conditions under study are the "low-stakes" environment of the current NAG administration and a higher stakes environment typified by many state assessment programs. The results suggest that in comparison to a "moderate-stakes" testing environment NAEP does not seriously underestimate achievement levels. However, the results cannot lead to the conclusion that student achievement is unrelated to testing stakes. Nor can one conclude that substantially raising the stakes of NAEP would not be accompanied by an increase in achievement scores.
This article summarizes research on the effect of testing on student achievement as found in English-language sources, comprising several hundred studies conducted between 1910 and 2010. Among quantitative studies, mean effect sizes range from a moderate d ≈ 0.55 to a fairly large d ≈ 0.88, depending on the way effects are aggregated or effect sizes are adjusted for study artifacts. Testing with feedback produces the strongest positive effect on achievement. Adding stakes or frequency also strongly and positively affects achievement. Survey studies produce effect sizes above 1.0. Ninety-three percent of qualitative studies analyzed also reported positive effects.
Since 2001, considerations of school reform have been dominated by performance-based accountability. No Child Left Behind (NCLB) has changed the way policymakers and educators talk about education, look at educational performance, and think about educational challenges. Nonetheless, NCLB and the state accountability systems it has spawned have been subjected to little careful scrutiny. This article discusses four recent research contributions and considers how they might inform policymaking on accountability. While scholarly scrutiny will not necessarily settle debates, it can help yield more constructive and informed decisions. In particular, research can clarify the actual consequences of policy decisions; highlight and refine approaches that may be more reliable, stable, and effective than those in use; flag the unanticipated or overlooked effects of design decisions; and ensure that both policymakers and the public are aware of the costs and benefits of accountability.
Consistent with the current emphasis on performance-based accountability in K-12 education, several states and a few local districts have introduced school-based incentive programs. This paper provides one of the few evaluations of the effects of such programs on student outcomes. Using a panel data set for schools in large Texas cities, it measures the gains in student performance in Dallas relative to those in other cities. It finds positive and relatively large effects for Hispanic and white seventh graders, but not for black students. Potentially positive effects also emerge for drop-out rates and principal turnover rates.[JEL I20]