April 10th, 2014
This post is the first I have ever done about an administrative matter at this university, but this merits it. I have always been passionate about teaching. I have long experimented with ways to engage my students, moving from the “sage on the stage” to the “guide on the side” approach. As a department chair, I championed the creation of a skills development room for “guide on the side” instruction in the new management building. When my colleagues’ course evaluations arrived in my office, I immediately read through them all – including student comments – and wrote my own comments on the evaluation form and occasionally met with colleagues who needed support. I’ve noticed that my two successors continued this practice.
Student course evaluations are a vital part of teaching, providing feedback that can help us improve. The University of Toronto has now moved from paper to online course evaluations. Having done research on e-government, this looks to me like a familiar Gov 1.0 reform intended to reduce organizational cost and hopefully improve user convenience. Sometimes these reforms work out (online vehicle registration) and sometimes they are met with resistance (the recent closing of Veterans’ Affairs offices, replaced by Service Canada’s online case management).
The major problem with the online course evaluation system is that, despite providing a two week window for students to complete the form, participation rates are substantially lower than the paper-based approach. In that approach, the instructor designated a fifteen minute period during one of the last three classes of the semester, gave out the questionnaires, and absented herself while the forms were completed and then collected.
At UTSC, the Course Evaluation Team affiliated with the Centre for Teaching and Learning has provided all sorts of advice about how instructors can boost the participation rate (giving the students 15 minutes in class to go online and complete the evaluations, telling students how much evaluations matter, showing prepared powerpoints, and emailing helpful reminders). Despite all this effort, it does not appear that the participation rate is increasing to anything comparable to the rate for paper evaluations. When all this social marketing is not achieving its objective, one must ask why. I think the problem is that students are being given too long to complete the questionnaire. The end of term is a busy time and, despite all the reminders, many never get around to it. And those who do are either those who were very satisfied or very dissatisfied.
The virtue of the old paper-based system was its immediacy. One chance to complete the evaluation – now or never. Interestingly, the Rotman MBA program has combined the immediacy of the paper-based system with the cost-reduction of the online system. As in the old system, evaluations take place over a fifteen minute period determined by the instructor. Students are encouraged to bring their laptops and go online when the electronic window is opened. Participation rates are comparable to those in the paper-based system.
While this is my primary concern with the online course evaluation, there are three others. The online system uses a five-point scale, rather than the seven-point scale of the paper-based system. I understand that the literature concludes that seven-point Likert scales aren’t a great improvement on five-point Likert scales. As a chair, however, I’ve seen hundreds of evaluations and very few go much below the mid-point (perhaps because students are co-producers of their own education and a very low score is an implicit self-criticism). So a seven point scale is really a three point scale (between 4 and 7). Thus, with this skewed distribution of responses, a five point scale becomes a two point scale (between 3 and 5). Bringing back the seven point scale would provide more variance.
Second, as mentioned previously, as chair I read the evaluations carefully before returning them to faculty members. The online system does not provide a step for review by the chair. I believe it should.
Third, by virtue of being entirely online, with no need for Scantrons, the university should be able to return the evaluations to faculty members faster than the paper-based evaluations. This does not appear to be the case.
I should finally note that the online course evaluations currently have one advantage over the paper-based system. Because all the information is gathered online, it is possible to do data analysis. Thus, faculty members now can see the average departmental and faculty evaluations to compare against their own, information the paper-based system never provided.
Online course evaluations, like many other instances of moving in-person or paper-based processes online, have the potential to increase efficiency, reduce cost, and provide more information. The U of T’s online course evaluation system, I regret to say, has not yet realized that potential.