Time for Feds To Come Clean on ‘Nation’s Report Card’ Tests

Printed from: https://newbostonpost.com/2018/04/10/time-for-feds-to-come-clean-on-nations-report-card-tests/

Quis custodiet ipsos custodes?, a Latin phrase found in the work of the Roman poet Juvenal, is commonly used to refer to the problem of how one monitors the actions of persons in positions of power. It means literally:  Who will guard the guards? Example:  We don’t know if we have a problem with the validity of the results of the so-called “Nation’s Report Cards” or the extent of the problem (if there is one) because there is little transparency on the test development process.

Tests given by the National Assessment of Educational Progress have been funded by Congress since their inception in the late 1960s and early 1970s. Federal law since 2001 has required that all states give these tests, the results of which are often called “the nation’s report cards,” as part of No Child Left Behind. By law, they can be given only to a stratified random sample of students across each state in each subject, and since the early 1990s they have been given in two forms. The long-term tests began in the 1970s and were to remain unchanged so that trends could be detected.  The main tests began in the 1990s and could occasionally change to reflect changes in the curriculum.

NAEP tests are given at three levels (grade 4, grade 8, and upper high school) according to a schedule worked out in advance. We are told that the 2017 results, to be released in mid-April 2018, may reflect computer-based testing, together with an explanation of how the results of paper-and-pencil tests differ from those generated from student use of computers. We won’t know more until the release.

Given the growing dissatisfaction with the statewide tests aligned to Common Core’s standards, it is not surprising that many parents are concerned about the independence and integrity of the “nation’s report cards.” Common Core standards are built into four-year education plans submitted by all state departments of education — without state legislative or local school board approval — to the U.S. Department of Education for the 2016-2017 school year. Do these national-assessment tests reflect the knowledge and skills in each subject area that subject experts in each area agree should be tested at tested grade levels?  Do they reflect the extent to which important areas of knowledge or skills are tested at a grade or over the grades? Do they reflect in part some of Common Core’s “standards,” which by law these tests’ items are not supposed to do? At present, we do not know.

Concerns have been expressed about mathematics in particular because it is the language of science and the foundation of most technical areas of study today and because of the decades-long controversies over how it should be taught, and what should be taught, in kindergarten through 12th grade.  Several questions need to be answered by the new commissioner of the National Center for Education Statistics, the organization/agency that constructs National Assessment of Educational Progress tests.  It should be noted that policies influencing the tests must be approved by the National Assessment Governing Board, the group appointed by Congress to shape the policies of the organization that constructs the tests.

One question is whether all NAEP mathematics test items are reviewed by a small group of mathematicians or scientists as part of the process of developing the test. The consensus of subject matter experts on the quality of each subject’s test items at each tested level is necessary in order to establish the validity of the test’s content. It is not clear that this consensus has been available for NAEP math tests, or even sought since the findings of the National Validity Study of 2007, which was commissioned by the organization that puts together the test. In this study, five mathematicians, listed as “technical advisers,” and representing a range of viewpoints on teaching methods, had been asked by National Center for Education Statistics to examine math items used on National Assessment of Educational Progress tests. The mathematicians found many test items to be “marginal” or “flawed,” achieving a remarkable consensus in their judgments, and wondered how valid NAEP math tests results could be if they included deficient items.

In a lengthy response to the National Validity Study of 2007, the National Center for Education Statistics’ commissioner at the time, Mark Schneider, raised the possibility that his organization may not have included enough subject matter expertise in the test development process for the national-assessment math tests.  He noted that questions about the “mathematical quality” of the test items was “the most prominent finding” of the validity study, adding: “Although the current standing committee has always included at least one mathematician, and there are many mathematicians available at [Educational Testing Service], we may not have achieved the needed representation of mathematicians during item development.”

In other words, the then-head of the organization that put together the test implied that it was possible that insufficient expertise in mathematics went into the development of the mathematics test items.  His remarks further implied that the organization that constructs NAEP tests may not have included a sufficient number of subject matter experts as reviewers of test items in other subject areas, as well.

A second question is whether the new mathematics test items added to the 2017 tests (at grade 4, grade 8, and upper high school) were aligned to Common Core’s standards.  The National Center for Education Statistics adds some new test items in most if not all testing cycles, but by law, National Assessment of Educational Progress test items are not supposed to reflect any particular set of standards. They are supposed to reflect the structure and salient features of the subject or discipline being assessed as judged by content experts. 

Why these two questions now? People who follow education are anxiously awaiting the results of the 2017 NAEP test results, expected within a week or so.  Concerns have been expressed about whether the use of computer-based testing versus paper-and-pencil tests affected the validity of the results of math and reading tests. Many observers expect the National Center for Education Statistics to address that concern. But not enough people are asking about a much greater concern: whether test items for these tests were vetted by subject matter specialists before the tests were given to children.

Is it too much to hope that Congress may ask those overseeing the tests:  (1) to ensure that at least a handful of subject matter experts reviewed the test items used in each round of testing in each subject tested and will do so in the future; (2) to make public the names of the expert reviewers used for the 2017 tests and in the future; and (3) to assure the public that National Assessment of Educational Progress tests did not and will not include test items aligned to Common Core’s controversial (and sub-par) standards?