Impact and Value of International University Rankings

From this week's THE:
Measured, and found wanting more
International comparisons of universities still have their detractors, but the appetite for them continues to grow. Phil Baty traces their roots and looks at how they are increasing in number and quality, while Ellen Hazelkorn considers their impact and value

Ellen's original article is available at http://arrow.dit.ie/cserart/2
Hazelkorn, E. (2009) 'Rankings and the Battle for World-Class Excellence: Institutional Strategies and Policy Choices', Higher Education Management and Policy, 21 (1)

Impact Factors, Citations and the h-index

As impact factors, citation analyses and the 'h-index' are used increasingly to measure research impact (see last week's post on university rankings), it is worth posting a reminder of what these are and of how the measurements are made.

Journal Impact Factors
Journal impact factors provide an objective measurement for differentiating between journals by measuring the frequency with which the average article in a journal has been cited in a particular year. An impact factor of 1.5 for a journal would mean that, on average, articles published in that journal during the previous two years have been cited 1.5 times. In the University of Limerick you can find a journal's impact factor in ISI's Journal Citation Reports (see the library list of databases).

Citation Analysis
A citation analysis measures the number of times an author, article, journal or institution is cited in academic literature. In the University of Limerick a citation analysis can be done in ISI's Web of Science (see the library list of databases). Do a search for the author, giving 'Univ Limerick' as the address. At the top right of your results list click 'Create Citation Report'. The citation report will provide graphs and statistics of the author's publication and citation trend, including the average number of times an item has been cited.

h-index
Included in the Web of Science citation report, the h-index reflects both the number of publications and the number of citations per publication. A h-index of h signifies that the author has published h papers each of which has been cited at least h times. So, for example, a h-index of 4 signifies that the author has published 4 papers each of which has been cited at least 4 times. The author may in fact have published 5 or even 100 papers, but papers 5 -100 have less than 4 citations and are therefore discounted. For further information on the h-index see http://en.wikipedia.org/wiki/Hirsch_number

Rankings using the measures above are based on the premise that if an academic or institution shows good citation metrics, it is very likely that the academic or institution has made a significant impact on the academic world. However, it is important to note that the reverse is not necessarily true and that an academic or institution with weak citation metrics may, for example, be working in a small field, publishing in a language other than English, or publishing mainly in books.

University rankings don't tell the whole story

Prof Ellen Hazelkorn, Director of Research and Enterprise at Dublin Institute of Technology and leader of the Higher Education Policy Research Unit (HEPRU), argues in this Irish Times article that too much focus on global rankings means that we risk transforming our higher education system to conform to metrics designed for other purposes.

Professor Hazelkorn looks at the various rankings, including the Shanghai Jiao Tong Academic Ranking of World Universities (SJT), the Times QS World University Ranking and the Taiwan Performance Ranking of Scientific Papers for Research Universities, and concludes that sometimes “global rankings focus on what is easily measured rather than measuring what counts”.

Stressing that “it is the total investment that is important”, she recommends that Ireland could usefully look to Australia and to Norway for models which recognise and reward excellence wherever it occurs, developing benchmarks which reflect our societal and innovation needs and assessing performance accordingly.

Links to Rankings information
Shanghai Jiao Tong Academic Ranking of World Universities (SJT)
Times QS World University Ranking
Webometrics – Ranking Web of World Universities
Wikipedia – College and University Rankings

Australia - the Excellence in Research for Australia (ERA) initiative
Norway - the Stjerno Commission

Link to the full Irish Times article