Beyond the Impact Factor

In a recent editorial, Katz and McGee (2019) discussed the impact factor of the Oncology Nursing Forum (ONF) and other journals, particularly the influencing effect of open access availability. For years, articles have been published denouncing the overreliance on any single measure of a journal’s quality or influence (Coleman, 2007; Magnus, 2013). There are numerous other mechanisms beyond the impact factor for assessing the citation-related metrics of journals and articles.

Citation Metrics

One additional metric is the Scopus CiteScore (https://bit.ly/2RVZdEj). As of January 2019, ONF has a CiteScore of 1.59, calculated by counting the citations received in 2017 to documents published in 2014, 2015, or 2016 and dividing this by the number of documents published in 2014, 2015, and 2016. Scopus is particularly useful for examining the citation metrics of individual articles. The ONF article with the most citations (459 citations) in Scopus is “The Revised Piper Fatigue Scale: Psychometric Evaluation in Women With Breast Cancer” by Piper et al. (1998). Table 1 lists the three ONF articles with the most citations in Scopus as of January 2019.

Another metric source is Google Scholar, which shows the same Piper et al. (1998) article as having 897 citations as of January 2019. Of note, Google Scholar has duplicate citations and, overall, lacks the precision and accuracy of databases such as Scopus. However, Google Scholar might include a citing source that is not identified by Scopus and other databases, so it is advisable to use as wide a variety of sources as possible when examining the citation counts of articles.

The FAQ page for Eigenfactor® (https://bit.ly/2UqXGmd) provides definitions of the Eigenfactor score and article influence score. Conklin and Oermann (2017) define the Eigenfactor score as “the number of times in the past five years that articles from the journal were cited in the Journal Citation Reports. The Eigenfactor score considers which journals have contributed these citations and removes journal self-citations” (p. 1). As of January 2019, the ONF Eigenfactor score was 66 and article influence score was 78, ranking 8th out of 118 journals in the nursing category.

Altmetrics

The altmetrics of an article indicate the number of times it is viewed, downloaded, and discussed on social media, by news outlets, and via other sources. It appears to be growing in importance as a measure of an article’s reach, as evidenced by some universities incorporating altmetrics into the tenure process—the more traditional approach was to focus on the impact factors of the journals that published faculty’s articles. The article’s dissemination is illustrated in the Altmetric “donut,” which provides a visualization of its attention from news outlets, tweets, Facebook, Wikipedia, and more. There is a necessary distinction between the general concept of altmetrics (lowercase “a”) and the organization Altmetric (capital “A”) responsible for the donut illustration. Other organizations provide altmetrics data, such as the PlumX product from Plum Analytics.

As of January 2019, the ONF article with the highest Altmetric attention score is “Establishing Priorities for Oncology Nursing Research: Nurse and Patient Collaboration” by Cox, Arber, Gallagher, MacKenzie, and Ream (2017), with a score of 95 (see Figure 1). This means that the article was covered by 10 news outlets; for example, a United Press International story (https://bit.ly/2TlqVa3) summarizes it, includes quotes from one of the authors, and contains text about the article being published in ONF (with links to the article on the ONF website.) It also shows the tweets and Facebook posts that have linked to the article, among them, the following tweet from @healthscisurrey (“Tweets From the School of Health Sciences, University of Surrey” at https://bit.ly/2WrOIXM):

Patients add value when setting priorities for Oncology research in the UK. See @AnnaClareCox’s Delphi study here: https://t.co/XQKLIT6xeu

Altmetric also tracks the number of readers the article has on Mendeley, a free Elsevier product that serves as a reference manager and academic social network for discovering and organizing research and collaborating with other researchers online. This ONF article is at https://bit.ly/2G2WPoX.

Other interesting uses of altmetrics for ONF are seeing where articles are referenced in patents, policy documents, and even Wikipedia and Reddit. For example, the ONF article “Oral Transmucosal Fentanyl Citrate for Cancer Breakthrough Pain: A Review” by Gordon (2006) is referenced by 17 patents, including “US8865743B2 Small Volume Oral Transmucosal Dosage Forms Containing Sufentanil for Treatment of Pain” at https://patents.google.com/patent/US8865743. In addition, the ONF article “The Role of Oncology Nursing to Ensure Quality Care for Cancer Survivors: A Report Commissioned by the National Cancer Policy Board and Institute of Medicine” by Ferrell, Virani, Smith, and Juarez (2003) is referenced in the National Academies Press publication Delivering High-Quality Cancer Care: Charting a New Course for a System in Crisis at https://bit.ly/2BanqMK.

Other Metrics

Unrelated to citation metrics, page views are a simple way to gauge an article’s popularity. Since ONF switched to a new online platform in 2015 (this does not incorporate the page views on previous platforms), the article with the highest number of page views is the “2016 Updated American Society of Clinical Oncology/Oncology Nursing Society Chemotherapy Administration Safety Standards, Including Standards for Pediatric Oncology” by Neuss et al. (2017), which, as of January 2019, has 41,541 page views according to Google Analytics.

Although it is not a formal numeric metric, ONF articles as information resources warrant inclusion in any discussion of value. Many libraries have years’ worth of print and/or online ONF holdings, and these articles are often consulted when their corresponding citations are retrieved in literature searches on PubMed, CINAHL®, and other databases. ONF articles are frequently cited and excerpted in response to questions asked via the online ONS communities and ONS clinical questions service. For a question on whether the implementation of a distress screening tool results in improved patient care, the ONF article “A Distress Thermometer Intervention for Patients With Head and Neck Cancer” by van der Meulen, May, Koole, and Ros (2018) was suggested because of the following:

The aim of this study was to investigate the feasibility of the DT&PL+ intervention and its effectiveness on depressive symptoms (primary outcome), HRQOL, and fear of cancer recurrence in patients with HNC. The authors hypothesized that, one year after inclusion, patients with HNC in the intervention group would report fewer depressive symptoms, better HRQOL, fewer physical symptoms, and less fear of cancer recurrence than patients with HNC in the control group. (p. E15)

For a question asking if streptozocin is a vesicant or an irritant, the ONF article “Vesicant Extravasation Part II: Evidence-Based Management and Continuing Controversies” by Wickham, Engelking, Sauerland, and Corbi (2006) was recommended because streptozocin is listed as a vesicant in a table in that article.

Conclusion

Articles published in ONF are used in a variety of ways, not all of which can be neatly captured in a single numeric metric. And as Katz and McGee (2019) stated, the impact factor “is not a measure or commentary of an article’s quality” (p. 9). Oermann (2012) reinforces this point:

A study describing the effects on patient outcomes of hourly rounding on a unit may be an influential article to clinical practice, yet have few citations to it. Many articles intended for clinicians and that improve patient care may never be cited or will receive a small number of citations. (p. 299)

ONF’s most important contribution is stated in its mission of disseminating the findings from oncology nursing research and fostering the translation of research evidence to practice; ultimately, this connection to the nursing care of patients with cancer is what matters most, regardless of any citation counts.

About the Author(s)

Mark Vrabel, MLS, AHIP, ELS, is an information resources supervisor at the Oncology Nursing Society in Pittsburgh, PA. No other financial relationships to disclose. Vrabel can be reached at mvrabel@ons.org, with copy to ONFEditor@ons.org.