Arbitrator makes significant ruling on student questionnaires on courses and teaching

An arbitrator has ruled that student questionnaires on courses and teaching cannot be used to measure teaching effectiveness in promotion and tenure decisions at Ryerson University.

William Kaplan ruled in favour of the Ryerson University Faculty Association and ordered its faculty Collective Agreement be amended to reflect the decision. While he does find that student questionnaires on courses and teaching can continue to be used in the context of tenure and promotion decisions, they cannot be used for the purposes of measuring teaching effectiveness for promotion and tenure. Kaplan’s decision could set a strong precedent for other faculty associations working to discontinue the inappropriate use of student questionnaires.

When used as a means of providing formative feedback to teachers, student questionnaires can help teachers make improvements to their course and their teaching approach. However, the use of student questionnaires for career-shaping decisions raises serious concerns about how faculty teaching is evaluated. Questionnaire results are skewed by a long list of factors, including personal characteristics such as race, gender, accent, age and attractiveness, and course characteristics including class size, quantitative versus humanities, traditional teaching versus innovative pedagogy, etc.

“The lack of reliability of SET results is further complicated when SET results are reduced to averages and then compared with other faculty members, the Department, Faculty and the University,” Kaplan wrote. “The evidence is clear, cogent and compelling that averages establish nothing relevant or useful about teaching effectiveness. The use of averages is fundamentally and irreparably flawed. The only relevant metric is frequency distribution.”

He continued: “SETs may be ubiquitous, but this does not serve as a justification for over-reliance on a flawed tool.”

Kaplan therefore ordered the following:

  1. The collective agreement (Collective Agreement) at Ryerson University is to be amended to ensure that SET results are not used in measuring teaching effectiveness for promotion or tenure.
  2. The numerical rating system shall be replaced with an alphabetical one.
  3. Question 15 (the summative question) should be struck.
  4. The parties should agree upon an appropriate, user-friendly, intelligible and easily accessible mode of presentation of frequency distributions together with response rates.
  5. The parties must ensure that Deans, tenure committees, promotion committees, and any others charged with evaluating faculty are educated in the inherent and systemic biases in SETs.
  6. The parties shall develop governing guidelines to be incorporated into the CA.
  7. The parties shall establish a properly resourced joint committee with hard timelines to consider the current SET and the possible revision/addition to the questions that are asked, the provision and use of comments, the process and methodology for the administration of SET (paper, online, in class, scheduled time, data tracking of response rates).
  8. The online SET system for non-online courses shall be immediately discontinued for probationary faculty.