Blog

Our intern, Clay Johnson, recently posted a blog about the NCES Report linking the proficiency cut scores of the 50 state tests (plus the District of Columbia) onto the scale of the NAEP assessment. The report provided updated cut score estimates and noted changes of these estimates from prior NCES reports released in 2007 and 2005. As Clay observed in his blog post, the general findings from the NCES report reflected those from our own recently published State of Proficiency report.  Although the sets of rankings produced by the studies were not in perfect agreement (Spearman’s Rho coefficients for the four sets of rankings of states included in both studies ranged from a low of .32 for fourth grade math up to a high value of .67 for eighth grade reading), both studies found a similar general pattern of variation across states in what constitutes reading and mathematics proficiency standards:  some states set very easy proficiency standards that can be met by most students with reasonable effort, whereas other states set their standards at such high levels that only the most gifted students could reasonably be expected to meet them. 

Although the widespread adaptation of common content standards across states (i.e., The Common Core) doesn’t address the issue of varying proficiency standards across states, at least studies such as the NCES Report and our own State of Proficiency Report have helped to create awareness of the issue by making direct comparisons across states possible.  But there is another issue identified within these reports, which in my opinion, doesn’t receive nearly as much attention as it should.  This is the issue of differences in proficiency standards across grades within a single state.

Although the NCES report only examines fourth and eighth grades (because it uses the NAEP scale as its benchmark), the State of Proficiency Report looks at grades three through eight, mapping the proficiency standards at each of these six grades onto the scale of the NWEA math and reading assessments and expressing them as percentile ranks, relative to NWEA norms.  Doing so, one can easily see how much variation in difficulty exists across these grades within a single state, as shown in the figure below, which gives the reading and math proficiency standards for California (or examine other states by selecting from the dropdown list).

In California, the proficiency standards for math start out relatively easy in the third and fourth grades at the 39^th percentile, growing progressively more difficult up to eighth grade, which is set at the 61^st percentile.  This means that about 61% of a normative sample of third or fourth graders would be expected to perform at or above the “proficient” level, while only about 39% of a normative sample of eighth graders would be expected to meet proficiency standards. 

This notion that proficiency standards are set at different levels of relative difficulty across grades is not generally acknowledged.  When the media report, for example, that higher percentages of elementary than middle school students meet proficiency standards, the implication is that there are problems in the middle schools, not that the proficiency standards are too easy at the elementary grades.  Public perception may drive policymakers and school administrators to allocate resources to boost middle school test performance, when these students may in fact be doing just as well as the elementary students.

This very issue was explored within the State of Proficiency report to determine the extent to which differences in state-reported proficiency rates across grades might be attributable to differing proficiency standards.  A nationally representative sample of students was collected, and their test performance was evaluated as though they lived within every state.  In this way, it was possible to estimate the proficiency rates of that sample for every state.  Test performance for that sample was also evaluated against a “calibrated” standard in which the proficiency standards at every grade were equally difficult.  These analyses are illustrated in the figure below for California (or for other states by selecting from the dropdown list).

The blue line shows the performance of our random sample under current California proficiency standards.  As can be seen, approximately 65% of the third graders in the sample meet the math proficiency standard, while only 43% of eighth graders meet that standard.  To the naked eye, this certainly looks like a drop in proficiency of 22%.  However, if students in all grades were held to similar levels of accountability, then proficiency rates would remain nearly identical across all six grades, ranging from 45% at third grade to 43% at eighth grade.  Using such “calibrated” proficiency rates, it’s much easier to see how much (or whether) student performance has changed across grades, or how best to implement resources for school improvement.  In the case of the California example, this probably means that a much more global approach will be necessary to boost student proficiency rates beyond 43-45%.  Without considering the impact of calibration on proficiency rates, however, administrators and policymakers won’t have all the information they need to make the right decisions about school improvement and helping their students learn.