A system that has long been used in several universities across the country to evaluate teachers’ in-class performance has also continually caused doubts about its effectiveness and parity among OU faculty members.
OU professors and instructors said the class evaluation system employed at OU and other institutions can result in significant biases, and the numerical system used is not effective in telling them how to improve their teaching skills. Some professors have implemented their own solutions, while another group at OU is preparing for larger institutional change.
Michael Givel, a political science professor and president of the OU chapter of the American Association of University Professors, said he believes the current evaluation system is “more of a customer survey” than an effective professional development tool.
“Being a woman or (from an) underrepresented group or teaching a topic that is not in the comfort zone of some students can result in adverse ratings,” Givel said.
According to a study on assessment and evaluation in higher education, “biased student evaluations of teaching could disadvantage faculty from underrepresented minority groups or punish faculty members who teach unpopular required courses.”
Sarah Warmker, a Spanish instructor, also said the numerical part of the system — the one through five rankings — is likely to reflect biases students might have based on professors’ race, gender or age.
“My own belief is that using a numerical system to evaluate teaching effectiveness is ultimately more harmful than helpful, especially to people with marginalized identities. But the university is using that as a shortcut,” Warmker said. “Just the same way that grades are used as a shortcut, as a way to compare people to each other as if the playing fields were equal, but we all know that it really isn't.”
Warmker said she has worked closely with the OU Multicultural Advancement Committee as a graduate student to analyze class evaluations, and a subgroup in the committee analyzed course evaluations as part of diversity considerations, curriculum and student experiences.
“When I was working with the Multicultural Advancement Committee, even the provost at the time — Kyle Harper— acknowledged that the course evaluation system wasn't really a reliable or a scientific way of evaluating performance and that the way the university really used it was to identify the outliers,” Warmker said.
According to a transcript provided by Warmker of the Multicultural Advancement Committee meeting on April 2019 discussing student teaching evaluations, Harper said four to five years of research has raised concerns about discrimination and bias — both unintentional and intentional — in the class evaluations, specifically among faculty who hold marginalized identities.
“There are genuine reasons to be concerned that bias does affect evaluation, and so one of the things we’re looking at is how do you correct (and) control that,” Harper said in the meeting. “I really fundamentally believe that student teaching evaluations should never be the only way we are evaluating a teacher, and that’s why we’ve pushed more peer evaluation, but understand that’s flawed too. None of these are perfect.”
As someone who “probably read more student teaching evaluations than anyone on campus,” Harper said the student teaching evaluations were flawed but still provided some useful insights.
“(Teaching evaluations) empower students. In an ideally functioning system, faculty are reading them, using them for self improvement. Nobody advocates (for) just getting rid of student teaching evaluations. It’s all about, 'How do you improve them? How (to) make them better?'” Harper said in the meeting. “It provides an important means of self assessment and self improvement for faculty, and it’s an anonymous voice for students who might not otherwise feel empowered if it were not anonymous to say.”
Givel said the anonymity of class evaluations is good in the sense that it allows for “unhindered surveys.” He said, however, if the survey seems to contain “controversial comments,” the anonymity allows for less accountability.
Harper also said context is important to interpret numbers as the size of classes can influence the scores those will receive. Small classes can get higher scores than big classes, major classes can get higher scores than required general education courses and electives can get higher scores than required courses, Harper said.
“Anybody who thinks that being in the 60th percentile versus being in the 50th percentile in a teaching evaluation is particularly meaningful, I think, is wrong. They are very useful at the extremes,” Harper said in the 2019 meeting. “Generally, the people who are regularly in the top five percent are generally the people who … students say, ‘This person really changed my life, this person spent a lot of time with me, this person was a mentor,’ and generally the people who are regularly in the bottom 10 to 20 percent, generally need some kind of professional development — there’s something flawed. So there’s valuable information.”
Warmker has implemented her own student evaluation survey in her classes. She said she doesn’t believe it is effective to ask students to fill out teachers’ evaluations when the class ends. Her survey is sent out at the beginning of her class, asking students about their good and bad experiences with language learning, and she then does a midterm evaluation “just to check in.”
“(Course evaluations) can be very influential in things like hiring processes,” Warmker said. “If I were to apply to teach somewhere else, sometimes other institutions will ask for copies of the course teaching evaluations. So, they can have a huge impact on someone's teaching career and can even get you put on probation (if) you have a low response rate.”
Warmker said it is “pretty mind-blowing” that OU has not updated the course evaluations during the pandemic” even though the learning situation has changed so much.”
In an email to The Daily, the Teaching Evaluation Working Group wrote it gathered answers in a collaborative effort to identify “some of the ways the traditional course survey system is in need of an update” and “recent efforts to improve the system.”
The Teaching Evaluation Working Group was established in January 2019 by the Office of the Provost and Faculty Senate for the OU Norman campus when both students and faculty expressed dissatisfaction with the current class evaluation system.
The group is made up of "experienced faculty and staff members" who have the "necessary expertise" and represent a range of disciplines. Those were selected to be part of the committee by the Faculty Senate Faculty Welfare Committee and OU Interim Senior Vice President and Provost Jill Irvine, according to OU Director of Media Relations, Kesha Keith.
According to the group, the current student evaluation — the Student Teaching Evaluation — has been in place for years and is widely used in colleges and universities across the country. Designed many years ago, little about the survey has changed in the last 20 years, the group wrote.
TEWG also found numerous studies demonstrating traditional STEs, such as the one used at OU, “are prone to bias and are not aligned with student learning.”
“Professors whose students learn the most and who are doing the work of adapting their courses to use techniques proven to enhance learning are often not getting the highest evaluations,” the group wrote in the email. “This is particularly true when that professor is a woman or is from a historically underrepresented group. The traditional STE is not designed to give instructors the kind of feedback they can use to improve their courses and is not well-suited to hybrid or online courses.”
Warmker and Givel also agreed that biases in student evaluations can happen based on the grade the student has received in the course. They said it is common for students to evaluate the class poorly regardless of instructional quality if they receive an undesired grade.
“Grades hold such a huge weight in students’ mind of how they feel about the class that sometimes they're not even really evaluating the teaching effectiveness or whether they learned, but they're really focused on whether they feel the grade they got was fair, and that there's a huge sense of injustice,” Warmker said. “So, I think just the same as there (are) equity problems in evaluations of professors, there (are) equity problems in the way that grades are assigned as well.”
Givel suggested the establishment of a “mid-course feedback” in which the results are only available to the instructor to consider in terms of teaching goals and tweaking areas of teaching for the class. He said the evaluations must focus on learning progress and skills by replacing “numerical ratings that could be biased” with qualitative feedback.
In the email, TEWG wrote it has created a new tool to gather student feedback — the Student Experience Survey — to address growing dissatisfaction with current teaching evaluations.
“The SES is designed to ask students concrete questions about what they experience in the classroom, (which) gives students a chance to reflect on the course and give more effective feedback, targeted in a way that gives instructors information they can use to assess their course and improve it,” the group said. “The ultimate goal of the TEWG is to use teaching evaluation to create a culture that encourages development of teaching skills and that rewards the use of evidence-based, effective teaching practices. “
The TEWG ran a pilot project in summer 2020 in over 500 course sections using an early draft of the new SES end-of-course survey in addition to the current STE so students and faculty could provide feedback with references to both. With the feedback received, the new pilot — which will take place at the end of the spring 2021 semester — has shown some departments across campus, such as the OU Extended Campus, the Biomedical Engineering, Drama, Health and Exercise Science, Library and Information studies departments, have opted to use the SES as their primary end-of-course survey.
According to TWEG, SES is different from the current class evaluations in the sense that it gives students a chance to reflect on their experience in the course and provides “targeted feedback” for instructors and professors to improve their courses.
SES was also designed to minimize the potential biases that have been observed in the traditional student teaching evaluations. TWEG wrote instead of asking students to evaluate or compare instructors, SES asks specific questions about student perceptions and experiences within the course.
“This gives students the opportunity to share their unique and valuable perspectives in areas where they are the experts — their personal learning experiences,” the group wrote.
Additionally, the new class evaluation system is designed to include different types of courses and modes of teaching such as in-person, hybrid, online classes, discussion, lecture, laboratory, field, and performance-based courses. TWEG wrote the SES asks the same core questions for all courses, unlike the eValuate survey questions which vary significantly between colleges and types of courses.
“The faculty response so far has been overwhelmingly positive,” the group said. “Our colleagues have helped us craft the questions so that they are broadly applicable across disciplines and mode of delivery. They have also suggested places where the survey could be shortened or simplified.”
TWEG wrote OU plans to implement the SES campuswide in the fall 2021 semester.
Ultimately, Warmker said the university needs to fix deep-rooted equity issues at OU before problems in the evaluation system can be effectively addressed.
“I think the course evaluations are really just a symptom of deeper problems,” Warmker said. “Until the deeper problems of equity are addressed both for students and for the people who do the work of teaching in the university, there will always be people (who) will never feel that the course evaluation system is fair because the conditions that people are teaching and learning in are not fair.”