Blog

The Gates Foundation recently published their latest reports from the Measures of Effective Teaching Project.  One of their claims in the study is that a combination of structured classroom observation, student surveys, and student achievement gains provide the best estimates of teacher effectiveness.  

Jay Greene’s critique of this study is really incisive and deserves your attention.  Greene points out, correctly I might add, that the study results actually show that student achievement gains, taken alone, predicted teacher value-added results much better than structured observations and student surveys.  More importantly, student achievement gains actually lost predictive power when combined with the observations and surveys.  In other words, one cannot claim that structured classroom observation and surveys actually measure teacher effectiveness better than using tests alone.  In addition, the classroom observation instruments alone showed relatively weak correlations to the participating teachers’ underlying value-added results.  None of the four protocols used showed correlations greater than .24 in mathematics and .20 in reading. 

As Greene notes, observational protocols are labor intensive.  For purposes of this project, between 17 to 25 hours of training were required to validate a rater on one instrument.   To get decent reliability statistics (.65), four lessons from a teacher had to be rated, each lesson by a different rater.  As Greene notes, this would require a considerable investment of time and money.   Given these limitations, one can fairly question whether these tools provide enough added accuracy to require their use in the teacher evaluation process.

Greene’s critique can be taken to imply, however, that these observational protocols have no role to play in evaluation.  Here I disagree. 

First, the primary purpose of the protocols is (and should be) to help teachers improve their teaching, and it’s difficult to improve teaching without someone observing instruction and providing good feedback.  The protocols show whether the teacher understands and can execute the mechanics of teaching, and a good protocol will also show where a teacher needs to improve.  We don’t seem to have a problem with this concept in sports.  Football coaches spend, on average, as much time watching film as they spend on the field with players.  And I just spent $100 on golf lessons because I’m convinced that having an expert observe my performance would improve it (and based on the results, I was right).  So I’m not particularly troubled if the protocols don’t add much statistical power to an evaluation metric, because they weren’t really designed for that purpose.

Second, regardless of their statistical power, virtually all teacher evaluation systems require that low-rated teachers be given support and an opportunity to improve.  The observational protocols serve an essential purpose here, because they provide information about the teacher’s classroom practice that could be used to improve it.  Ideally, this data is used to help the teacher meet professional expectations, but if improvement fails to occur, the protocols also provide specific evidence of the teacher’s deficiencies that will make a termination more defensible.   This is what makes the protocols indispensable. 

Finally, there is a tendency to confuse the role of classroom observation with the role of the principal in the evaluation process.  All the observational protocols establish is whether the teacher can execute the mechanics of teaching on, in this case, four occasions.  They may show whether an educator is capable of teaching, but they don’t constitute compelling evidence of an educator’s competence over a complete school year.  An educator has to deliver competent instruction consistently.   That requires that the educator show up for work consistently, engage well with students, be prepared and focused, correct and return work in a timely fashion, and meet the professional obligations associated with teaching.  Evaluating these aspects of the teacher’s performance is the work of a principal, and in my opinion, the principal’s judgment as to how a teacher performs on these matters that should drive the evaluation.  Put another way, success on the protocols should not trump a principal’s judgment on job performance, because the principal is charged to judge the teacher’s entire body of work.