Measurement is Fundamental

Two seemingly-unrelated articles about academe recently crossed my radar; a colleague at MGSC forwarded an article on Defining Teaching Effectiveness by Maryellen Weimer, while Inside Higher Ed today ran an article on what scholarship "counts" for tenure and promotion in the digital humanities.  While the two articles seem to touch on very disparate aspects of the professoriate, they do have one big thing in common (besides, of course, both articles dealing with aspects of the faculty review portfolio): they both fundamentally are questions of measurement.  More importantly, they are issues of measurement that reflect the same sorts of decisions social scientists have to make every day in their work.

Let's tackle "teaching effectiveness."  Clearly, there are lots of ways to measure the effectiveness of teaching.  By far the most common (probably because it's cheap, easy, and doesn't require any actual effort by the consumer of the evaluation to produce) is the notorious student evaluation of teaching (SET), wherein students typically answer dozens of questions and administrators cherry-pick one or two to actually use.  If that weren't bad enough, the move to online evaluations—coupled with a bizarre belief that it's somehow unethical to make students evaluate a course as one of the requirements of taking the course in the first place—has led to enormous self-selection issues in response rates that (if we're honest) make any attempt to glean generalizable information from them about as pointless as using ratings from Ratemyprofessors.com to evaluate the quality of law schools.  It doesn't matter how good your instrument is if you have a 10% response rate.

Complaints about the SET process aside, though, it can be a useful entree to questions of measurement and sampling in a methods class.  Every student has been asked to complete these evaluations, so they come in with some familiarity with the process; if not, the process is easily grasped by simply presenting a copy of the instrument your institution uses.  Unlike when we measure most political science phenomena, SET instruments have multiple measures of the same underlying concept (teaching effectiveness), which can be used to illustrate the concepts of reliability and validity.  You also can discuss also how different respondents might have different degrees of motivation to participate (perhaps even tying in other situations where negative feelings motivate more participation and engagement than positive ones, such as midterm elections and responses to campaign advertising).  And you can challenge students to think about how the instrument could be redesigned (if at all) to better separate students' feelings about the course from their feelings about the instructor; if nothing else, it might give a subtle cue to your students to give you the benefit of the doubt when evaluating you at the end of the semester.

Comments

I love the idea of using the

I love the idea of using the student evaluation forms as a learning tool in a methods class.  There are any number of important topics that you could connect them too.  I wonder if engaging the instrument in this way will make students more or less likely to take them seriously in other courses.  

My expectation is that they'd

My expectation is that they'd probably take it a bit more seriously, but I could see an argument either way. And thanks for your comment!