Rankings and reality

Last month the Shanghai ranking kicked off the season. This week the QS World Universities Ranking was revealed and in two weeks the all new Times Higher Education ranking (THE) will be published. Ranking season also means discussions about the value of rankings and about their methodologies. Two points of critique are addressed here: the volatility of (some) rankings and the overemphasis of research in assessing universities’ performance.
 

Volatility and stability in international rankings

This year’s discussion has become particularly fierce (and, occasionally, nasty) after THE’s decision to part with consultancy agency QS and join forces with Thomson Reuters, a global research-data specialist. The previous joint THE/QS rankings usually received quite some media attention. This was not just because their methodology was heavily criticized (and rightly so), but also because it led to enormous fluctuations in the league tables from year to year. The critique has induced THE to join forces with Thomson. Meanwhile, QS continues to produce its own ranking.

Although the various rankings differed in their methodology, they all seemed to agree on two things: the hegemony of  United States universities and Harvard's undisputed leadership. This week’s QS rankings again showed the volatility of their methodology. For the first time, Cambridge beat Harvard and for the first time the top ten is not dominated by US universities. The top ten is now occupied by five US universities and five UK universities.

The Shanghai ranking, on the other hand, shows much less fluctuations. It is probably a better reflection of reality, but it is also less sensational and, as a result, attracts less media attention. The two graphs below clearly show the difference between the stable Shanghai rankings and the volatile QS rankings for a selection of four Dutch universities.

The graphs show the positions in the past six years for the four Dutch universities that are in the top 100 of the Shanghai and/or QS ranking. To illustrate the relative meaning of the absolute positions, the Shanghai rankings groups institutions above rank 100 (this also explains Erasmus University’s relatively steep drop in the 2006 ranking). Although Amsterdam has remained fairly stable in the rankings, Leiden and Utrecht show quite some fluctuation. Much more than its real quality would justify.

And if you think this is volatile, know that it can be much worse.  In a 2007 paper Simon Marginson lists dozens of cases of drops and rises of more than 50 positions (sometimes even up to 150 positions) within in a year's time. A case in point is the Universiti Malaya which went from 'close to world class' to 'a national shame' in only two years...

Download Simon Marginson's paper (357 kB)

It will be interesting to see in the coming years how the new THE/Thomson methodology will work out in this respect. The Times Higher published its methodology this week. While the QS ranking is bases on only 6 indicators (with a 50% weighting towards  reputational surveys), the new THE ranking looks at 13 indicators (grouped in five categories). Considering the higher number of indicators and the significantly lower weighting towards reputational surveys , the THE ranking is expected to fluctuate less than the QS ranking. Time will tell…
 

Do international rankings assess teaching quality?

Another frequently mentioned critique on the existing international rankings is that they put too much emphasis on assessing research and neglect universities’ teaching role. Since the new THE ranking more than doubled the number of indicators, it is likely that the assessment will correspond better with the complex mission of universities.

If we look at the new methodology this indeed seems to be the case. Teaching now accounts for 30% of the overall score and is based on 5 indicators. In the QS ranking, it was based on only 2 indicators (employer’s survey and staff-student ratio).

The 5 indicators are:

1. reputational survey on teaching (15%);
2. PhD awards per academic;
3. undergraduates admitted per academic (4.5%);
4. income per academic (2.25%);
5. PhD and Bachelor awards.

A closer look at these 5 indicators, however, raises the question of how much they are related to teaching.

First of all, we can question if a reputational survey really measures the quality of teaching or whether it is just another proxy for research reputation. Colleagues and peers around the world often do have some idea of the quality of research in other institutions, but is it likely that they can seriously evaluate the quality of teaching in other institutions? Apart from the institutions where they graduated or worked, it is unlikely that they can give a fair judgement about the quality of teaching in other institutions, let alone in institutions abroad.

Second, two other questionable indicators for the quality of teaching are the number of PhDs awarded and the number of PhD awards per academic. In the Netherlands, and in many other countries in continental European and elsewhere, this says much more about the research quality and the research intensity of an institution than about the teaching quality.

Third, the indicator ‘undergraduates admitted per academic’ seems the same as their old student-staff ratio indicator. Assuming here that a lower number is better, this again benefits research intensive institutions more than other institutions. Research intensive institutions employ a relatively high number of academics, but many of them will have research only contracts. Yet, thanks to this indicator they will still get a higher score for teaching quality.

Finally, ‘income per academic’ is also a dubious indicator. Assuming this concerns the average annual income of academics, there is no reason to believe that a higher salary leads to a higher quality of teaching. It could be argued that salaries are nowadays more related to research quality and productivity than to teaching quality. If ‘income per academic’ refers to the external financial resources that an academic attracts, it would even more be an indicator of research intensity.

At first glance the new THE ranking methodology seems to put more emphasis on teaching, but at closer look this is rather misleading. All this again shows how difficult it is to measure teaching quality. However, as long as the international rankings don’t sufficiently address teaching quality, they cannot fulfil their role as transparency instruments for international students.