Getting Smart About Measuring School Performance

Here at the C-SAIL blog, we’ve written previously about the role of power as a policy attribute of standards-based reform systems. Power is succinctly defined as “rewards and sanctions:”

A policy has power if educators are rewarded for behavior or outcomes consistent with the policy, or are penalized for behavior or outcomes inconsistent with the policy. Such rewards and sanctions can be tangible, such as an increase in pay linked to increased student test scores. Or, power can work through less tangible but still extrinsic rewards or penalties, such as public recognition of either good or poor performance.

The primary source of power in the current implementation of standards-based reform is through the accountability policies required under federal law (the Elementary and Secondary Education Act (ESEA)). The most recent version of ESEA, the Every Student Succeeds Act (ESSA), devolves greater control of the design of accountability to states, and there is a great opportunity for states to substantially revise accountability policies moving forward.

The federal Department of Education is currently in the process of issuing regulations to clarify the ESSA law for states. These regulations will have profound consequences for the kinds of accountability policies that are actually written and implemented under ESSA. It is in this context that I recently wrote an open letter to the Department of Education as part of their formal comment process. You can see the letter I wrote here. At the bottom is the list of signatories, which numbers close to 100, and includes former presidents of the American Educational Research Association, former Commissioners of The National Center for Educational Statistics and the Institute of Education Sciences, and more than a dozen K-12 educators and school district data and accountability officers. The letter was also covered by news media here, here, and here.

The crux of my letter (written with the help of my former student Stephani Wrabel) focuses on how states should measure school performance. Under No Child Left Behind (NCLB), schools were evaluated based on the percentage of their students scoring at or above a state-determined proficiency level on mathematics and English language arts tests. Research showed that this design led to negative unintended consequences: 

• It encouraged schools to focus their energies on students just below this proficiency cut score, since improving student achievement for already-proficient (or very-far-below-proficient) students would not help the schools’ accountability ratings.
• It is statistically a very poor measure of school performance because the relative rankings depend heavily on where the proficiency score is set.
• It is a very poor measure of achievement gaps—actual changes in achievement gaps can sometimes be opposite to changes identified by comparing proficiency rates, again because of where proficiency levels are set and the distribution of student achievement.

And there are more arguments, which I detail in the letter.

Based on the research evidence cited above, I argue in the letter that the Department of Education should interpret the law to allow states to use alternative measures of performance. In particular, I recommend one of two options:

1. The simple average of students’ scores on the assessment (what’s known as average scale scores)
2. A “proficiency index” that gives schools credit for performance all along the achievement distribution (for instance, 0 points for far below basic, 25 for below basic, 50 for basic, 75 for proficient, and 100 for advanced).

In the letter I recommend the first approach because it uses the most information available in the assessment data. However, both approaches would be far superior to percent proficient.

So why am I so concerned about such a seemingly esoteric design feature of accountability policy? In short, incentives matter. Educators understandably pay attention to the design of accountability systems. Given the stakes involved, which can sometimes include reconstituting schools, it is not surprising that educators would shape their instructional responses based on the incentives of the policy. Though the stakes under ESSA are likely to be somewhat lower than under NCLB, there is still a need for accountability policies to encourage good practices, not bad ones.

The letter, however, did not describe my ideal accountability system, which would be one based heavily on student achievement growth. That kind of policy would encourage schools to focus on growing ALL students, and it would be much fairer to schools that serve large proportions of low income and other historically underserved students. Unfortunately, such a policy would not be allowable under my read of ESSA statute, so I could not advocate it in the letter.

I sincerely hope that the Department of Education takes my letter seriously, because such a simple change could have profound effects on the incentives in ESSA. Others have written letters that make the same or similar points, so there will be ample voices behind these policy changes. Regardless, it is encouraging that so many researchers, educators, and advocates are interested in improving accountability policy for the next generation.