Maclean's survey flawed: Dal

Maclean’s magazine’s use of data collected from two national student surveys paints a false comparison of Canadian universities, says an analyst for Dalhousie University.

Canadian universities are incomparable because of significant variations in student population, class size and programs offered, says analyst Michael O’Sullivan. Photo Illustration: Aly Thomson

Maclean's magazine is misrepresenting data from two national surveys it uses to rank Canadian universities, a research analyst for Dalhousie University says.

The magazine published the results of two annual student surveys on Monday: the National Survey of Student Engagement and the Canadian University Survey Consortium. The surveys are commissioned by the universities and used to examine student satisfaction.

Maclean's uses the results to create a countrywide ranking of universities.

And it's "bogus."

That's what Michael O'Sullivan, senior analyst at Dalhousie's Office of Institutional Analysis and Research, calls it. O'Sullivan, whose office provides institutional research to Dalhousie's administration, strongly disagrees with Maclean's representation of the data.

"It is simply an inappropriate way to use the data," said O'Sullivan, who is also on the board of directors at the Canadian University Survey Consortium. "The problem with a ranking ... without some indication of the statistical significance of the differences among universities, is that it can create a false picture."

According to O'Sullivan, it forces readers to compare universities that, in actuality, are incomparable. He said the difference in how two students respond to the same question is not large enough to be statistically significant.

"Flaws"

The Consortium divides schools into three groups: small, medium, and large in accordance to student population. In 2009, 34 institutions participated, representing approximately 1,000 graduating students at each university. He said universities tend to compare themselves to institutions in the same group, but Maclean's lumps the schools into one lengthy list.

"Beyond that, Maclean's does not make any attempt to indicate how large of a difference it would have to be between university scores to make a real difference in the student experience."

Maclean's editors could not be reached for comment, but in the preamble of the survey, the magazine claims each ranking chart lists the universities in descending order of "achievement," and the questions are "the broadest and most representative of the student experience."

For example, the magazine uses the National Survey data to answer the question: "How would you evaluate your entire educational experience at this institution?
The top three schools rated by first-year students are:

  • the University of King's College, with 1,100 students
  • B.C's Trinity Western University, with 4,000 students
  • Queen's University, with 22,477 students

Founder of the National Survey, George D. Kuh, has dismissed Maclean's use of the data since the magazine began ranking in 2006.

In a letter to the editor of The Gauntlet, the University of Calgary's student newspaper, he explains how the rankings are flawed because they reduce the many dimensions of university life to a single number. Ranking Canadian schools is especially inaccurate because each university has a different mission, offers different programs, and varies in size, he said.

"Rankings may sell magazines but they do little to help the public understand what makes for a high-quality undergraduate experience," he said in the July 2006 article.

Survey and Consortium administrators have been expressing their views to the magazine for four years, O'Sullivan said.

But Maclean's is not budging.

"They're aware of our views," said O'Sullivan.

"Particularly valuable" information

All rankings aside, O'Sullivan said the information gathered from the surveys is extremely useful.

The National Survey focuses on five key areas:

  • level of academic challenge
  • student-faculty interaction
  • active and collaborative learning
  • enriching educational experience
  • supportive campus environment

Fourteen institutions representing 8,965 students participated in the 2009 survey.

The Consortium survey targets one of three student populations each year: first-year students, graduating students and all undergrads. This year, graduate students answered questions examining student issues and experiences.

O'Sullivan said this survey is "particularly valuable because our successful students can tell us quite a lot about their student experience during their time on campus."

The data is used by the universities to progressively improve the school and better student experience.

Although Dalhousie published the National Survey information, the university has not posted the Consortium results - yet.

But the Maclean's chart states Dalhousie "declined to make this information public," even though O'Sullivan says the information will eventually be posted on the website.

The surveys are made public by the individual universities and Maclean's obtains the data from school websites or directly from the university.

O'Sullivan says it's important that prospective and current students be able to see the data.

"I think it is a good thing for this information to be in the public realm," he said. "It is not a good thing that Maclean's, or anyone else uses it with dubious statistical underpinnings to rank universities."

 

Comments on this story are now closed