Sandford Borins

Sandford Borins, Ph.D.

Sandford Borins is a Professor of Management at the University of Toronto. He writes, blogs, and teaches about narrative, information technology, and innovation.

Learn More.


April 10th, 2014

How to Fix U of T’s Online Course Evaluation System

Education, Living Digitally

This post is the first I have ever done about an administrative matter at this university, but this merits it. I have always been passionate about teaching. I have long experimented with ways to engage my students, moving from the “sage on the stage” to the “guide on the side” approach. As a department chair, I championed the creation of a skills development room for “guide on the side” instruction in the new management building. When my colleagues’ course evaluations arrived in my office, I immediately read through them all – including student comments – and wrote my own comments on the evaluation form and occasionally met with colleagues who needed support. I’ve noticed that my two successors continued this practice.

Student course evaluations are a vital part of teaching, providing feedback that can help us improve. The University of Toronto has now moved from paper to online course evaluations. Having done research on e-government, this looks to me like a familiar Gov 1.0 reform intended to reduce organizational cost and hopefully improve user convenience. Sometimes these reforms work out (online vehicle registration) and sometimes they are met with resistance (the recent closing of Veterans’ Affairs offices, replaced by Service Canada’s online case management).

The major problem with the online course evaluation system is that, despite providing a two week window for students to complete the form, participation rates are substantially lower than the paper-based approach. In that approach, the instructor designated a fifteen minute period during one of the last three classes of the semester, gave out the questionnaires, and absented herself while the forms were completed and then collected.

At UTSC, the Course Evaluation Team affiliated with the Centre for Teaching and Learning has provided all sorts of advice about how instructors can boost the participation rate (giving the students 15 minutes in class to go online and complete the evaluations, telling students how much evaluations matter, showing prepared powerpoints, and emailing helpful reminders). Despite all this effort, it does not appear that the participation rate is increasing to anything comparable to the rate for paper evaluations. When all this social marketing is not achieving its objective, one must ask why. I think the problem is that students are being given too long to complete the questionnaire. The end of term is a busy time and, despite all the reminders, many never get around to it. And those who do are either those who were very satisfied or very dissatisfied.

The virtue of the old paper-based system was its immediacy. One chance to complete the evaluation – now or never. Interestingly, the Rotman MBA program has combined the immediacy of the paper-based system with the cost-reduction of the online system. As in the old system, evaluations take place over a fifteen minute period determined by the instructor. Students are encouraged to bring their laptops and go online when the electronic window is opened. Participation rates are comparable to those in the paper-based system.

While this is my primary concern with the online course evaluation, there are three others. The online system uses a five-point scale, rather than the seven-point scale of the paper-based system. I understand that the literature concludes that seven-point Likert scales aren’t a great improvement on five-point Likert scales. As a chair, however, I’ve seen hundreds of evaluations and very few go much below the mid-point (perhaps because students are co-producers of their own education and a very low score is an implicit self-criticism). So a seven point scale is really a three point scale (between 4 and 7). Thus, with this skewed distribution of responses, a five point scale becomes a two point scale (between 3 and 5). Bringing back the seven point scale would provide more variance.

Second, as mentioned previously, as chair I read the evaluations carefully before returning them to faculty members. The online system does not provide a step for review by the chair. I believe it should.

Third, by virtue of being entirely online, with no need for Scantrons, the university should be able to return the evaluations to faculty members faster than the paper-based evaluations. This does not appear to be the case.

I should finally note that the online course evaluations currently have one advantage over the paper-based system. Because all the information is gathered online, it is possible to do data analysis. Thus, faculty members now can see the average departmental and faculty evaluations to compare against their own, information the paper-based system never provided.

Online course evaluations, like many other instances of moving in-person or paper-based processes online, have the potential to increase efficiency, reduce cost, and provide more information. The U of T’s online course evaluation system, I regret to say, has not yet realized that potential.


April 4th, 2014

Guide for Innovative Public Servants Just Published

Government, Innovation

The IBM Center for the Business of Government has just published my report The Persistence of Innovation in Government: A Guide for Innovative Public Servants. In it, I discuss both shifts and continuities in public sector innovation over the last two decades, using applications to the Harvard Kennedy School’s innovation awards as well as sources from other countries. The report presents my major findings as well as implications for practitioners and researchers.

Here is a link to its page on the IBM Center website: Links on the top left corner of the page enable you to download the report in its entirety or order free hard copies.

The report itself is based on my book, The Persistence of Innovation in Government, which will be published next month by Brookings. I will post about the book when it becomes available.



March 29th, 2014

Who Will Rally the Anti-Ford Nation?


Watching the first Toronto mayoral debate, I was impressed by the ability of Mayor Ford to stay on message, and I was disappointed by the inability of the other candidates, especially during the discussion of leadership, to hold him accountable for his behavior. What we heard were euphemistic criticisms of the “circus at City Hall” or of Ford being a “poor role model.” With softball criticisms like this from the other candidates, it wasn’t hard for Ford to stay on message. The polls after each segment of the debate clearly indicated that Ford still has strong support from the members of Ford Nation who were texting their approval.

If the Toronto mayoralty election was a race involving a small number of well-financed candidates, we would likely see Ford’s opponent running attack ads to keep reminding the public of Ford’s behavior, making it harder for him to keep claiming that it was simply yesterday’s news (“rewind, rewind, rewind,” as he put it). The candidate herself could then take the high road and not attack Ford personally.

This is not the way the campaign is currently constituted. The four challengers (Chow, Tory, Stintz, and Soknacki) in the debate were all taking the proverbial high road, likely hoping that one of the others would launch a sustained attack on Ford. None did.

While I don’t have public opinion polls to prove it, I’m convinced that there is an Anti-Ford Nation in Toronto, a large group of people who are fed up with Ford’s antics and drama and who want to see any candidate other than Ford elected. Hence they are more concerned about the leadership issue than about other issues such as transit or taxes, and they are waiting to vote for whichever candidate is most likely to defeat Ford.

The question for the four major challengers is how to win the support of the Anti-Ford Nation. It seems logical to me that the best way to do this would be to launch the most aggressive attack on Ford’s leadership, as a way of drawing the sharpest possible contrast with his or her own integrity.

At the second debate, held the following night at Ryerson University, John Tory came closest to realizing this, and dialed up the tone of his attack on Ford. Quoting from the Globe and Mail, he said “it is not acceptable to have a mayor who shows up late and sometimes doesn’t show up at all … and most troubling and unacceptable of all, a mayor who has admitted multiple and continuing relationships with convicted criminals and gang-types.”

The two major candidates who are now furthest behind and therefore have the least to lose by attacking Ford are David Soknacki and Karen Stintz. Soknacki has a very comprehensive and thoughtful position on mayoral transparency and ethics on his website (see my post of Jan. 21, 2014) and should have summarized and referred to it in the debate. Stintz, regrettably has decided that the election is “not a referendum on Rob Ford” and has urged voters to focus on the other policy issues. My analysis is that she’s absolutely wrong. (Personal disclosure: I had one encounter with Ms. Stintz when she was running for councilor that convinced me she tends to operate on transmit more than on receive.)

A sustained attack on Ford would make reference to his absenteeism, alcoholism, drug use, relationships with criminals, homophobia, racism, and sexism. A sustained attack on Ford would ask why he is even running for mayor rather than entering rehab. And such an attack would raise the considerable possibility that Ford’s future would be radically transformed if he were indicted for a criminal offense.

This recitation of Ford’s failings would establish a contrast with a challenger’s discussion of how he or she would do the work of mayor with focus, honour, and integrity.

As much as Ford attempts to deal with his drunkenness, drug-taking, and consorting with criminals as “rewind, rewind, rewind” it is highly likely that there will be more such incidents happening during the campaign. A challenger who continues to attack that behavior will find that these incidents will powerfully underscore the validity of those attacks.

My conclusion: there is an Anti-Ford Nation waiting to be led, and it is not yet clear which of the major challengers has the boldness to take up the leadership of that nation.


March 22nd, 2014

Big Data for Budget Discussions


I notice it has been over a month since my last post. I’ve been very busy with my teaching – in particular the budget simulation I’ll discuss in this post – and with my research. I’ve been reviewing the copy-editing and page proofs for my next book (The Persistence of Innovation in Government) and its accompanying monograph (The Persistence of Innovation in Government: A Guide for Innovative Public Servants). Both will be published in the next month or two.

The budget simulation involved students representing twelve major Ontario program ministries which were required to submit proposals for new programs or enhancements to existing programs to be funded from a hypothetical $3 billion policy reserve. The proposals were discussed in a cabinet meeting and the final decisions made by the premier and finance minister.

I’ve run simulations that involve allocating either a policy reserve for new programs or a global budget cut for many years. The availability of spending data (estimates, public accounts, results based plans) for the federal and Ontario governments has led me to attempt to make these exercises as realistic as possible. However, the students have always had to confront the problem of inadequate data. The estimates and public accounts present government spending at a very high level of aggregation. Results-based plans provide some detail about a variety of programs, but are not comprehensive. It is therefore difficult for students to determine what existing programs actually cost, which makes it hard for them to determine what either scaling up or scaling down a program would cost.

In this era of big data, there is no longer any excuse for inadequate data about public spending. Open government websites make available a variety of data bases collected by government. Government websites make available various politically salient aspects of government spending, such as travel and entertainment expenses for senior executives (federal) and salaries over $100,000 for the public service and broader public sector (Ontario).

It is time to apply to apply this approach to public spending. This would involve providing comprehensive data about, for example, government program contributions to organizations (such as grants to hospitals, firms, or cultural organizations) or entitlement spending by region or overhead spending by program. There are numerous ways the data could be organized while still protecting the privacy of individuals.

This data could be used by citizens, including students in public management course, to debate budget cuts or spending increases. What would it cost if the Ontario government wanted to decrease class size in elementary school? What would the federal government save if it wanted to close down a number of consulates? Better data would enable citizens to debate these questions. Discussions of government spending tend to be conducted secretly because detailed data are not made available to the public. Making the data available, which is well within the technological and organizational capability of government, would expand the discussion to include the public.

Governments run consultations in the run-up to releasing a budget. The consultations are of limited value because the dialogue is one way (the federal government asking for ideas to be emailed to the Finance Department) or because simulations are conducted at far too high a level of aggregation (Ontario’s recent consultations). Providing big data on government spending could change this.

Returning to the budget exercise, here are some proposals made by the students representing program departments, and accepted by the students representing the premier and finance minister of Ontario, for new or enhanced programs.

• Increase the Department of Education’s going green program from a pilot program to eventually encompass all Ontario schools (25 percent in the first year)

• Increase funding for research in curricular design for science, technology, and math

• Increase funding for the RIDE program

• Train 50,000 more health care workers and increase funding for community based specialty clinics, to reduce hospital use

• Provide larger subsidies to encourage drivers to trade in older cars for newer more energy efficient cars in particular hybrids

• Begin building access roads to the Ring of Fire

• Increase support for the Second Career program, especially for apprenticeships and skilled trades

• Increase support for entrepreneurs in “green” businesses and tourism

Eventually, a new Ontario budget will be released. Perhaps some of these proposals – or something like them – will be included in it.


February 13th, 2014

A Clash of Grumpy Old Men or a Multi-faceted Teachable Moment?

Government, Politics

Occasionally an event occurs that has so many aspects that it can be used to represent most of the issues raised in a university course. The conflict that surfaced between Veterans Affairs Minister Fantino and Canadian war veterans brings out many of the issues that I discuss in the public management course I am teaching this term. (I showed Terry Milewski’s story that ran on the CBC, which is available on YouTube under “Fantino meeting disrespects veterans.)

The confrontation came about over a decision by the Harper Government to shut Veterans Affairs offices in eight locations (Kelowna, Saskatoon, Brandon, Thunder Bay, Windsor, Sydney, Charlottetown, and Cornerbrook) where demand apparently has declined, and consolidate the department’s activities with Service Canada. The issue is whether integrated service delivery actually improves service. In some cases it does; for example, Service Ontario offices allow users to renew their health card, driver’s license, and license plate in one visit, rather than two or three visits to different government offices (see my post of April 7, 2011). But aren’t the needs of veterans so complex (case management involving physical infirmities, psychological disabilities, and various transfer payments) that they continue to merit dedicated service?

At his meeting with the veterans, Minister Fantino steadfastly defended the government’s decision, as cabinet solidarity would dictate. Pressure in favor of the decision undoubtedly came from the Department of Finance and Treasury Board Subcommittee on Government Administration, both charged with a cost-saving agenda. One wonders how Fantino and his department responded to the proposals to close their offices when they were still in the discussion stage. Did they have counter-proposals that might have been more acceptable to their constituency, such as collocating Veterans’ Affairs offices with Service Canada offices and ensuring that veterans, at least initially, would have a designated service queue, rather than having to be served in a common queue with all other clients? Developing a counter-proposal would have been the work of the department.

Perhaps Fantino might have taken a more aggressive stance personally, and threatened to resign from cabinet if the offices were closed. I think his threat would have been very credible. At age 71, Fantino is already receiving a generous pension from his service as top cop in Toronto and Ontario. He won the heavily Italian constituency Vaughan from the Liberals for the first time in recent memory. The Conservative Party of Canada probably needs Julian Fantino more than he needs the Conservative Party.

Fantino was late for his meeting with the veterans because he was at a cabinet committee meeting. Why didn’t he simply leave the cabinet committee meeting? Fantino’s handling of the meeting with veterans wasn’t very effective. Of course they informed the media in advance and were looking for a confrontation. And Fantino, by adopting a posture of standing rather than sitting and cutting off a spokesman for the sin of finger-pointing, provided it. Despite the bad hand cabinet had dealt him, a minister with better social skills and a less confrontational demeanor could have handled the situation more effectively.

Another thing I mentioned to the students was the veterans’ media strategy. Elderly men with lapels covered in decorations for bravery and distinguished service getting emotional at a press conference will always win public sympathy.

Prime Minister Harper defended the government’s cost-cutting policy in the House of Commons, as would be expected given the government’s over-riding objective of reducing expenditures to balance the budget, as well as his own uncompromising personality. This doesn’t mean Harper will indefinitely stand by this minister.

What will Julian Fantino’s political fate be? He demonstrated himself to be a clumsy, inept, arrogant, and unappealing defender of government policy. Perhaps in a few months Harper will have a don-like discussion with Fantino, telling him not to expect to remain in cabinet if the Conservatives are re-elected. The sub-text of that message would be that it is time for Fantino to take one for the team, and announce his retirement, so the party can find a more presentable candidate to contest Vaughan.

The Veterans’ Affairs story is now off the front pages, but I expect it will continue to simmer in the months ahead. Inserting in the budget initiatives to improve online services for veterans, to give them hiring preference for public service jobs, and to increase funeral and burial payments are all efforts to mollify what had been one of the government’s steadfastly supportive constituencies. We’ll see how this story plays out on both the service delivery and the political levels as the Conservatives prepare for the next election.