How to Fix U of T’s Online Course Evaluation System

This post is the first I have ever done about an administrative matter at this university, but this merits it. I have always been passionate about teaching. I have long experimented with ways to engage my students, moving from the “sage on the stage” to the “guide on the side” approach. As a department chair, I championed the creation of a skills development room for “guide on the side” instruction in the new management building. When my colleagues’ course evaluations arrived in my office, I immediately read through them all – including student comments – and wrote my own comments on the evaluation form and occasionally met with colleagues who needed support. I’ve noticed that my two successors continued this practice.

Student course evaluations are a vital part of teaching, providing feedback that can help us improve. The University of Toronto has now moved from paper to online course evaluations. Having done research on e-government, this looks to me like a familiar Gov 1.0 reform intended to reduce organizational cost and hopefully improve user convenience. Sometimes these reforms work out (online vehicle registration) and sometimes they are met with resistance (the recent closing of Veterans’ Affairs offices, replaced by Service Canada’s online case management).

The major problem with the online course evaluation system is that, despite providing a two week window for students to complete the form, participation rates are substantially lower than the paper-based approach. In that approach, the instructor designated a fifteen minute period during one of the last three classes of the semester, gave out the questionnaires, and absented herself while the forms were completed and then collected.

At UTSC, the Course Evaluation Team affiliated with the Centre for Teaching and Learning has provided all sorts of advice about how instructors can boost the participation rate (giving the students 15 minutes in class to go online and complete the evaluations, telling students how much evaluations matter, showing prepared powerpoints, and emailing helpful reminders). Despite all this effort, it does not appear that the participation rate is increasing to anything comparable to the rate for paper evaluations. When all this social marketing is not achieving its objective, one must ask why. I think the problem is that students are being given too long to complete the questionnaire. The end of term is a busy time and, despite all the reminders, many never get around to it. And those who do are either those who were very satisfied or very dissatisfied.

The virtue of the old paper-based system was its immediacy. One chance to complete the evaluation – now or never. Interestingly, the Rotman MBA program has combined the immediacy of the paper-based system with the cost-reduction of the online system. As in the old system, evaluations take place over a fifteen minute period determined by the instructor. Students are encouraged to bring their laptops and go online when the electronic window is opened. Participation rates are comparable to those in the paper-based system.

While this is my primary concern with the online course evaluation, there are three others. The online system uses a five-point scale, rather than the seven-point scale of the paper-based system. I understand that the literature concludes that seven-point Likert scales aren’t a great improvement on five-point Likert scales. As a chair, however, I’ve seen hundreds of evaluations and very few go much below the mid-point (perhaps because students are co-producers of their own education and a very low score is an implicit self-criticism). So a seven point scale is really a three point scale (between 4 and 7). Thus, with this skewed distribution of responses, a five point scale becomes a two point scale (between 3 and 5). Bringing back the seven point scale would provide more variance.

Second, as mentioned previously, as chair I read the evaluations carefully before returning them to faculty members. The online system does not provide a step for review by the chair. I believe it should.

Third, by virtue of being entirely online, with no need for Scantrons, the university should be able to return the evaluations to faculty members faster than the paper-based evaluations. This does not appear to be the case.

I should finally note that the online course evaluations currently have one advantage over the paper-based system. Because all the information is gathered online, it is possible to do data analysis. Thus, faculty members now can see the average departmental and faculty evaluations to compare against their own, information the paper-based system never provided.

Online course evaluations, like many other instances of moving in-person or paper-based processes online, have the potential to increase efficiency, reduce cost, and provide more information. The U of T’s online course evaluation system, I regret to say, has not yet realized that potential.

 

5 comments

  1. Hi Professor Borins,

    I’m a former student of yours (C42 course in Winter 2006), and I’m currently attending Wilfrid Laurier University for a Masters of Arts in Community Music program. They still use the paper forms, and I have suggested that electronic forms be used.

    Why is it not a REQUIREMENT that students IMMEDIATELY fill out these forms? Virtually all students have smartphones, and if this online form was smartphone friendly, probably participation rates will show up a lot higher!

  2. Another possible problem here is perception of confidentiality. Word on the student street seems to be that some don’t trust the promise of anonymity/confidentiality. As it becomes increasingly clear that online privacy is an illusion, one spinoff is a declining trust in the confidentiality of online surveys.

  3. Good write up Professor Borins,

    I’ve always found that by course evaluation time I am often drowned in the many other responsibilities and projects I have to undertake to prepare for exams and final assignments. Without a sense of urgency, it is very easy to overlook course evaluations.

    I’ve found the paper based system a lot more effective as it gives the student the time needed to complete the evaluation. One of my professors this semester actually gave us the class time to fill out the evaluations on our electronic devices in class and it was clear that this action alone sparked many students to do the evaluation then and there in class.

    As a student I find these evaluations very important as it allows me to give my 2 cents on the quality of education provided by a professor for the semester, and hopefully better the experience for future classes.

    Bernard’s point on a cell phone friendly test is a very good one. Perhaps even an app for students to install that could ping reminders as well would increase response rates.

  4. Why not just give a bonus 1%? You’d be surprised at what students would do just to increase their grade even by a little.
    Also I understand what you’re saying, it’s like voting. Most people don’t really believe their response will make much of a difference. I only started doing course evaluations because one of my professors, Wanda Restivo, actually mentioned in the first lecture that her evaluations said she spoke too fast and to let her know to slow down. Honestly, I think just saying something like that has an impact showing maybe students can make a difference in a learning atmosphere even if it is small.

Leave a Reply

Your email address will not be published. Required fields are marked *