Citation Analysis - how to ...

All researchers will be familiar with calls to measure or prove your impact through citation analyses. Whether you agree or disagree with such measures of quality, these Step-by-step guides to citation analysis for UL researchers using Web of Science (ISI) and Publish or Perish (Google Scholar) will be useful.

Guide to using Web of Science
Guide to using Publish or Perish

Journals
If the Web of Science does not cover the journals or books in which you publish (check the Master List here), you can apply to have a journal included. Click here for selection criteria and an application form.

Conference Proceedings
For those disciplines where Conference Proceedings are primary publications, the recent inclusion of Conference Proceedings in Web of Science should improve citations.

Books
Books are not covered by Web of Science, nor are there plans to begin coverage of books. Those researchers who publish mainly in books may be interested in this post suggesting alternative measures for assessing impact in the AHSS.

Publish or Perish , a Google Scholar citation analysis tool from Anne-Wil Harzing, can be useful where Web of Science results do not appear to give a fair reflection of your research output.

For more information on Citation Analysis, the h-index and Journal Impact Factors see the library webpage (www.ul.ie/library) at Supporting Research -> Research Publication & Dissemination here, or go directly to the site here.

The articles linked to in the first paragraph above are:

Agree
Garfield, Eugene (2006) The History and Meaning of the Journal Impact Factor. JAMA 295: 90-93.

Disagree
The PLoS Medicine Editors (2006) The Impact Factor Game. PLoS Med 3(6): e291. doi:10.1371/journal.pmed.0030291.

Assessing Research Impact in the Social Sciences and the Humanities

“If we wish to resist the mechanical imposition of standardised citation measures then we need to suggest reasonable grounds for exempting certain fields from metric assessment and propose alternatives.”

So writes Professor Tom Lodge, Assistant Dean Research AHSS, UL, in a memorandum which urges Social Sciences and Humanities researchers to publish, where possible, in formats that lend themselves to comparable, if not statistical, indications of impact. Professor Lodge goes on to outline some of the alternative measures that might be used to indicate the standing and quality of published work in AHSS, including:

Books
o Consider the status of the publisher, perhaps compiling lists of top ten publishers in various fields
o Pay attention to review coverage, noting the standing of the reviewing journals as well as the content of the reviews
o Distinguish between refereed and non-refereed books
o Consider a book’s publishing history e.g. reprints and second subsequent editions may be considered to signal impact
o Consider a book’s sales figures, allowing for adjustments where a publication is in a very specialised field

Journals not indexed by Web of Science
o Consider using Publish or Perish, a citation analysis tool which analyses Google Scholar
o Refer to the European Reference Index for the Humanities (ERIH) for journal ratings
o Gather information from publishers

Other
o Consider work that shapes public policy debates
o Consider the status of conferences, and ranking of conference participation

Professor Lodge’s full memorandum can be accessed here, or on the library webpage (www.ul.ie/library) under Supporting Research -> Research Publication and Dissemination -> Citation Analysis.

Impact Factors, Citations and the h-index

As impact factors, citation analyses and the 'h-index' are used increasingly to measure research impact (see last week's post on university rankings), it is worth posting a reminder of what these are and of how the measurements are made.

Journal Impact Factors
Journal impact factors provide an objective measurement for differentiating between journals by measuring the frequency with which the average article in a journal has been cited in a particular year. An impact factor of 1.5 for a journal would mean that, on average, articles published in that journal during the previous two years have been cited 1.5 times. In the University of Limerick you can find a journal's impact factor in ISI's Journal Citation Reports (see the library list of databases).

Citation Analysis
A citation analysis measures the number of times an author, article, journal or institution is cited in academic literature. In the University of Limerick a citation analysis can be done in ISI's Web of Science (see the library list of databases). Do a search for the author, giving 'Univ Limerick' as the address. At the top right of your results list click 'Create Citation Report'. The citation report will provide graphs and statistics of the author's publication and citation trend, including the average number of times an item has been cited.

h-index
Included in the Web of Science citation report, the h-index reflects both the number of publications and the number of citations per publication. A h-index of h signifies that the author has published h papers each of which has been cited at least h times. So, for example, a h-index of 4 signifies that the author has published 4 papers each of which has been cited at least 4 times. The author may in fact have published 5 or even 100 papers, but papers 5 -100 have less than 4 citations and are therefore discounted. For further information on the h-index see http://en.wikipedia.org/wiki/Hirsch_number

Rankings using the measures above are based on the premise that if an academic or institution shows good citation metrics, it is very likely that the academic or institution has made a significant impact on the academic world. However, it is important to note that the reverse is not necessarily true and that an academic or institution with weak citation metrics may, for example, be working in a small field, publishing in a language other than English, or publishing mainly in books.