The problem with measuring professor quality

Professors James West and Scott Carrell published an article last month in the Journal of Political Economy: Does Professor Quality Matter? Evidence from Random Assignment of Students to Professors. The article has actually been out for a few years, but this is the first time I came across it. Inside Higher Ed has a review of the study from 2008.

The authors looked at student scores and professor evaluations at the U.S. Air Force Academy from 1997 to 2007 and, in their own words, found this:

Results show that there are statistically significant and sizable differences in student achievement across introductory course professors in both contemporaneous and follow-on course achievement. However, our results indicate that professors who excel at promoting contemporaneous student achievement, on average, harm the subsequent performance of their students in more advanced classes. Academic rank, teaching experience, and terminal degree status of professors are negatively correlated with contemporaneous value-added but positively correlated with follow-on course value-added. Hence, students of less experienced instructors who do not possess a doctorate perform significantly better in the contemporaneous course but perform worse in the follow-on related curriculum.

Student evaluations are positively correlated with contemporaneous professor value-added and negatively correlated with follow-on student achievement. That is, students appear to reward higher grades in the introductory course but punish professors who increase deep learning (introductory course professor value-added in follow-on courses). Since many U.S. colleges and universities use student evaluations as a measurement of teaching quality for academic promotion and tenure decisions, this latter finding draws into question the value and accuracy of this practice.

To sum-up, the study found:

1) Students will score better in their intro courses when taught by less experienced professors, but they will do more poorly in subsequent courses.

2) Students will rate their professors higher when they get better grades in their intro courses.

In other words, if you teach a very rigorous course and do a really good job at preparing your students for success once they leave your classroom, you are likely to be punished for it with lower teacher evaluations. And if you make your class easy and everyone gets an A, you'll be rewarded with great evaluations. I suppose you still may be punished later with angry emails from students who can't pass their subsequent courses.

The study certainly draws into question how much importance we should place in rating professors based on their teacher evaluations.