Altmetrics: An Overview and Evaluation

Abstract

Purpose
The purpose of this paper is to provide an overview and critique of altmetrics, an understudied yet increasingly important arena of study for scholars, academics, and professional researchers.

Design/methodology/approach
The paper is organized into six parts: the first defines altmetrics; the second examines how altmetrics work; the third presents multiple typologies under which altmetrics can be classified and studied; the fourth details the technological capabilities of altmetrics; the fifth presents a critical evaluation of the “pros and cons” of altmetrics; and, the sixth outlines some directions for future and ongoing research.

Findings
The conclusions detail the strengths and limitations of altmetrics and point toward avenues for continued research and development.

Originality/value
This paper is among the first to provide a substantive review and evaluation of altmetrics for academics to consider when adopting, utilizing, and researching these tools.

Full text

Altmetrics: an overview
and evaluation
Ann E. Williams
Department of Communication, Georgia State University, Atlanta, Georgia, USA
Abstract
Purpose –The purpose of this paper is to provide an overview and critique of altmetrics, an understudied
yet increasingly important arena of study for scholars, academics, and professional researchers.
Design/methodology/approach –The paper is organized into six parts: the first defines altmetrics; the
second examines how altmetrics work; the third presents multiple typologies under which altmetrics can be
classified and studied; the fourth details the technological capabilities of altmetrics; the fifth presents a
critical evaluation of the “pros and cons”of altmetrics; and, the sixth outlines some directions for future and
ongoing research.
Findings –The conclusions detail the strengths and limitations of altmetrics and point toward avenues for
continued research and development.
Originality/value –This paper is among the first to provide a substantive review and evaluation of
altmetrics for academics to consider when adopting, utilizing, and researching these tools.
Keywords Altmetrics, Research impact, Academic networks
Paper type Research paper
Altmetrics: an overview
The primary purpose of this paper is to provide a description, overview, and evaluation of
altmetrics, an understudied yet increasingly important arena of study for scholars,
academics, and professional researchers. The paper is organized into six parts: the first
defines altmetrics and clarifies the concept of altmetrics in its various forms; the second
examines how altmetrics work; the third presents multiple typologies under which
altmetrics can be classified and studied; the fourth details the technological capabilities of
altmetrics; the fifth presents a critical evaluation of the “pros and cons”of altmetrics; and,
the sixth outlines some directions for future and ongoing research.
What are altmetrics?
Altmetrics are measurements of how people interact with a given scholarly work. They aim
to measure web-driven scholarly interactions, such as how often research is tweeted,
blogged about, or bookmarked (Howard, 2012; Robinson-García et al., 2014). Altmetrics.org
and Altmetric.com are web-based sites that promote the use of altmetrics.
Altmetrics.org is a site created, developed, and maintained by scholars and app designers
who share a commitment to “the creation and study of new metrics based on the social web for
analyzing, and informing scholarship”(Priem et al., 2010; http://altmetrics.org/about).
Altmetrics.org is concerned with the promotion of altmetric apps (i.e. ImpactStory,
ReaderMeter, ScienceCard, PLoS Impact Explorer, PaperCritic, and Crowdometer).
Altmeric.com is a commercial website, partnered with major publishers, which functions
as an open tool and data provider to supply qualitative and quantitative data that
complements traditional, citation-based measurements. Altmetric.com is concerned with the
promotion and circulation of its products in connection with major academic publishers,
institutions, and funders (e.g. Taylor & Francis, Wiley, The London School of Economics,
and the Smithsonian). Online Information Review
Vol. 41 No. 3, 2017
pp. 311-317
© Emerald Publishing Limited
1468-4527
DOI 10.1108/OIR-10-2016-0294
Received 5 October 2016
Revised 5 October 2016
Accepted 3 March 2017
The current issue and full text archive of this journal is available on Emerald Insight at:
www.emeraldinsight.com/1468-4527.htm
The author would like to thank Seifu Adem for his research assistance.
311
Altmetrics: an
overview and
evaluation

The rise of altmetrics
Technological changes of the 1990s and mid-2000s, characterized by the development of the
social web and internet-based social networking, enabled librarians to make many scholarly
works more openly and widely accessible to researchers and higher education communities
(Dutta, 2016). The creation and popularization of altmetrics –both the measures and the
websites –is a result of the growth of communication technology, especially social networking
sites such as Facebook and Twitter. As social networking provided new opportunities for
scholars to disseminate their research, new methods for capturing and calculating the
networked impact of scholarly publications became increasingly important (Dutta, 2016).
Proponents of altmetrics identify several factors that necessitated their emergence and will
precipitate their continued use. According to Galligan and Dyas-Correia (2013), one factor is the
limitation of existing measures of social, public, and/or “real-world”impact of research. For
example, traditional measures such as bibliometrics that measure peer-review, citation count, and
journal impact factors measure only the most academically significant and theoretically relevant
material from the huge volume of scholarly literature produced (Galligan and Dyas-Correia, 2013,
p. 56). A primary factor underlying the creation and continuance of altmetrics is that traditional
citation-based measures focus solely on journals and articles, but do not account for other work
outputs such as blogs and datasets. Altmetrics, in contrast, fill this void by measuring the impact
of journal articles through social media activity, while also accounting for other forms of
significant research output that fall outside the parameters of traditional peer-reviewed
publications. In this way, altmetrics enable the discovery of new information about research
impact that was previously difficult to obtain; and, they allow researchers to discern the impact of
their work at a faster rate than traditional metrics, such as citation counts and journal impact
factors that accrue more slowly. Some even suggest that the benefits of altmetrics may be far
greater than the benefits of other metrics. Most notably, altmetrics allow for valuable forms of
crowdsourcing that tap the immediate value of research within networked environments. Galligan
and Dyas-Correia (2013) note this trend, suggesting that through crowdsourcing an article’s
impact can “almost immediately be assessed by multiple bookmarks and conversations”(p. 57).
How do altmetrics work?
Supported by digital science, altmetrics aggregate information from different sources. Via
Altmetric.com, these include peer reviews, references on Wikipedia, public policy documents,
discussions on research blogs, mainstream media coverage, bookmarks on reference managers
like Mendeley, and social media. According to Melero (2015), Altmetric.com gathers data from
three main sources: social media, traditional media, and online reference managers such as
Mendeley. While Altmetric.com does not collect data from all online platforms, they do draw
from a wide array of sources, including blogs, news, Reddit, Facebook, Google Plus, Pinterest,
Twitter, Stack Exchange, CiteULike, Connotea, Mendeley, F1000, YouTube, LinkedIn Groups,
Research Highlights, and miscellaneous others (Robinson-García et al., 2014).
Typologies/classifications of altmetrics
As measurement tools, altmetrics are classified based on the function they provide and the type
of engagement users have with a given research output. For example, Robinson-García et al.
(2014) categorize different types of altmetrics by their primary functions: discussions, mentions,
readers, reviews, video, and citations. Sources that host discussions include blogs, news, Reddit,
and Stack Exchange. Sites such as Facebook, Twitter, Google Plus, Pinterest, and LinkedIn
Groups categorize mentions. Outlets such as CiteUlike, Connotea, and F1000 provide reviews.
YouTube illustrates impact through videos. And, Research Highlights provides full citations.
In extension of typologies based on functionality, others have conceptualized altmetrics
in terms of use (Wouters and Costa, 2012) and engagement (Lin and Fenner, 2013).
The following two tables summarize classifications by Lin and Fenner (2013) and Wouters
and Costas (2012), respectively (Tables I and II).
312
OIR
41,3

What are the technological capabilities of altmetrics?
Almetric.com assigns scores to an article by calculating how often the work is mentioned on
different media platforms. The popularity of the article is, therefore, based on how often it is
referenced in these sources. In addition to frequency of mentions, the altmetrics projected
via Altmetric.com include a record of attention, a measure of dissemination, and an indicator
of influence and impact. As a record of attention, Altmetric.com provides information on the
reach of a scholarly work, i.e., how many people discuss the piece of research. As a measure
of dissemination, Altmetric.com maps the location of the mention (where), and the reason
(why) an article has been shared and discussed. As an indicator of influence and impact,
Altmetric.com also provides a vehicle to capture how research can influence society-at-large.
In these ways, Altmetric.com has unique capabilities to measure the impact of different
research outputs, in terms of usage (downloads and views), peer-review (expert opinion),
citations, storage, links, bookmarks, and conversations.
Key functionalities
In order to track and collate the amount of attention a scholarly output receives, Altmetric.com
examines three essential components: an output ( journal article, dataset, etc.); an identifier
attached to the output (DOI, RePEc, etc.); and mentions in a source. Once the algorithm collects
mentions of an article, the site displays the information on its display page.
The display page also has several functionalities that provide information about an article.
The following are descriptions of the key functionalities that appear on the display page:
(1) The donut and attention score: this provides information on the types and the
amount of attention the research output has received.
Type of engagement Description
Viewed Accessing the article online
Saved Saving articles in online bibliography managers which helps researchers to organize papers
Discussed Discussion of the research described in an article, ranging from a short comment
shared on Twitter to more in-depth comments on blog postings
Recommended Endorsing the research article via a platform such as an online recommendation
channel
Cited Formation of a citation to an article published in a scientific journal
Source: Lin and Fenner (2013)
Table I.
Lin and Fenner’s
engagement-based
classification
Uses Descriptions
Diversity of channel analyzed Altmetrics enable the user to analyze different types of materials such
as books, blogs, Facebook postings, or tweets
Speed of acquiring/retrieving data Unlike traditional citation-based metrics, altmerics are instantly
available for analysis
Openness of method Altmetric data is open to download and free to use
The ability to measure impact beyond
the scholarly realm
Altmetric’s accessibility to different groups of readers such as
researchers, professionals, students, and interested publics helps
research escape the judgment of the scientific readers
Source: Wouters and Costas (2012)
Table II.
Wouters and
Costa’s use-based
classification
313
Altmetrics: an
overview and
evaluation

(2) Summary counts: this shows how many people from each source type,
e.g., Facebook, Twitter, F1000, etc., are mentioned or shared the work.
(3) Bibliographic details: this provides information on the names of authors, titles, dates
of publication, and other identifiers.
(4) Browse the original mentions: this function enables users to get access to sources
that mention the primary research, i.e., links to news stories, blogs, etc.
(5) Signup for alerts: this asks users to register in order to receive e-mail alerts
regarding the articles they select.
(6) Access the published research: this function enables users to get access to the
research output.
The details page also provides demographics and rankings as well as a map of
the geographical locations of users that illustrates the amount of attention a
research article receives in comparison with other research outputs published within a
similar time period and topical domain. The following screenshot of the display page
shows how Altmetric.com summarizes the attention a piece of research receives from
audiences (Figure 1).
Figure 1.
Altmetric.com
data map
314
OIR
41,3

The pros and cons of altmetrics
Pros
Although altmetrics are not a replacement for traditional measures in discerning the impact of
research, they do complement traditional measures. Speed is an asset of altmetrics that
traditional measures lack. Altmetrics enable users to have a quick view of the impact of a
research output (Melero, 2015). They allow users to get fast information about how many
times an article is mentioned or discussed by people in different sources. In addition to the
quick view of a research output’s impact, altmetrics also enable the swift dissemination of an
article. Due the speed at which people use social media, such as Twitter, it is easier to follow
and organize references of scholarly work right after they are published (Melero, 2015).
Altmetrics are also advantageous in that they have a wide-range of applications that help
track researchers’scholarly outputs, including data sharing, software, and presentations.
Furthermore, altmerics provide opportunities that researchers cannot get through other
measures, including discussion of works in progress and unpublished articles.
Cons
Although altmetrics are a useful data source, there are some limitations as well.
First, they are only complementary to, not a replacement for, traditional data such as
citation-based measurements.
Second, altmetrics can be compromised by subversive means or gaming, i.e., “the practice
of unscrupulously exploiting a system or set of data in order to produce results that fit a user’s
desired outcome”(Roemer and Borchardt, 2015, p. 20). This concern arises in environments
where data can be artificially manipulated. For example, one can get a “Like”on Facebook
from close friends, family members, etc. to promote his/her work, which may reflect a form of
personal or public impact but may not be a valid measure of scholarly impact.
The third limitation of altmetrics concerns the lack of a correlation between bibliometric and
altmetrics data (Roemer and Borchardt, 2015). Unlike biblometrics, there is no conclusive research
evidence that indicates a correlation between altmetric indicators and citation indicators.
Fourth, Altmetric.com’s inclusion of data from public social media like Facebook and
Twitter is a potential problem. According to Roemer and Borchardt (2015):
This concern leads to what is perhaps an even more relevant criticism of the inclusion of metrics
from non-academic-peer networks –that networks primarily populated by members of the general
public are much less likely to be interested in esoteric fields of research than in research that
connects to popular topics of discussion like climate change.
A fifth limitation is that Altmetric.com does not currently includes all possible sources in
which a scholarly work is mentioned and may omit or misidentify scientific research
(Robinson-García et al., 2014). For example, while Altmetric.com gathers data from Twitter,
it does not include mentions in Tumblr.
A sixth limitation is that Altmetric.com’s circulation data only includes impact scores for
articles published in English. For example, while Altmetric.com gathers mentions on
Facebook, it does not collect mentions on Spanish Tuneti (Robinson-García et al., 2014).
The seventh issue with Altmetic.com is related to its representativeness. For instance,
Robinson-García et al. observed that almost 95.5 percent of data gathered are derived from
only five sources, namely, Twitter, Mendeley, Facebook, CiteULike, and blogs.
The eighth limitation of Altmeric.com is a lack of clarity in the definition and
interpretation of what the metrics mean and what they do. For instance, it is difficult to
claim that mentions, recommendations by experts, reader counts, likes, and citations on
Twitter, F1000, Mendeley, Facebook, and blog posts have a common and universally
understood meaning (Haustein, 2016).
Table III summarizes some of the limitations, i.e., “Cons,”of altmetrics.
315
Altmetrics: an
overview and
evaluation

Directions for future research
This research report provides fundamental conceptualizations and critiques of altmetrics
that open the door for continued exploration and research in this domain. Two areas that are
of particular and immediate importance include:
(1) Technical issues. Altmetric.com, and the apps available via Altmetrics.org, do not
currently have the capability of collecting data on scholarly outputs produced in
languages other than English. In the future, design elements of these sites could be
enhanced by the inclusion of data from a broader range of sources and the creation
of apps that gather data from multilingual texts.
(2) Theoretical issues. Since the study of altmetrics is a new field, with development ongoing,
there is not a definitive theory-based explanation as to the mechanisms underlying their
emergence and continued growth. In the future, theory-driven approaches to studying
altmetrics will be of value to academic and scholarly communities.
In connection with the foundational conceptualizations of altmetrics presented in this study,
research that explores academics’adoption and use of altmetrics over-time will be essential
to uncovering the long-term impact and value that alt-measurements bring to scholarly
communities. From a user perspective, the advantages of altmetrics identified in this report
can be leveraged to heighten the dissemination and reach of academic research, particularly
in domains that provide research to the general public. From a design perspective, the
current limitations of altmetrics should be considered as new and improved tools are
developed and unveiled. The future of altmetrics is a fruitful area of study for designers,
academics, and researchers to continue to explore.
Cons Description
Not citation-based Altmetrics are only complementary to traditional citation metrics and do not
replace citation-based data such as bibliometrics
Gaming Data can be manipulated to fit a user’s desired outcome
Lack of significant correlation
with bibliometric data
There is no conclusive research evidence that documents a correlation
between altmetric indicators and citation-based indicators
Inclusion of public social media General publics may be less interested in academic research outputs and
more interested in popular topics
Lack of common definitions It is difficult to define activities such as mentions on Twitter, “likes”on Facebook,
and recommendations by experts on F1000 as sharing similar meaning
Heterogeneity of social media
platforms and users’
motivations
The nature of social media platforms such as Facebook, Twitter, and F1000,
etc., host a wide array of users, with different motivations and use behaviors,
that may not be directly comparable and/or uniformly impactful
Lack of conceptual
frameworks and theories
Scholars have yet to fully theorize and conceptualize altmetrics
Data quality Unlike other measures such as bibliometrics, where data can be triangulated,
the data in altmetrics are dynamic, in that they can be deleted or altered, and
may therefore lack consistency, accuracy, and replicability
Lack of inclusiveness Altmetrics do not include data from all digital media platforms
Language bias Altmetrics.org only collects data on research that is written in English. For
example, while they collect data on Facebook, they don’t collect mentions on
Spanish Tuneti
Table III.
The limitations
of altmetrics
316
OIR
41,3

References
Dutta, B. (2016), “Altmetricmanifestocompletesfiveyears(2010-2015)”,Current Science,Vol.110No.1,p.17.
Galligan, F. and Dyas-Correia, S. (2013), “Altmetrics: rethinking the way we measure”,Serials Review,
Vol. 39 No. 1, pp. 56-61.
Haustein, S. (2016), “Grand challenges in altmetrics: heterogeneity, data quality and dependencies”,
Scientometrics, Vol. 108 No. 1, pp. 413-423, doi: 10.1007/s11192-016-1910-9.
Howard, J. (2012), “Scholars seek better ways to track impact online”, The Chronicle of Higher Education,
January 29, available at: www.chronicle.com/article/As-Scholarship-Goes-Digital/130482/
(accessed October 1, 2016).
Lin, J. and Fenner, M. (2013), “Altmetrics in evolution: defining and redefining the ontology of article-
level metrics”,Information Standards Quarterly, Vol. 25 No. 2, pp. 20-26.
Melero, R. (2015), “Altmetrics –a complement to conventional metrics”,Biochemia Medica, Vol. 25
No. 2, pp. 152-160.
Priem, J., Taraborelli, D., Groth, P. and Neylon, C. (2010), “Altmetrics: a manifesto”, available at:
http://altmetrics.org/manifesto (accessed October 1, 2016).
Robinson-García, N., Torres-Salinas, D., Zahedi, Z. and Costas, R. (2014), “New data, new possibilities:
exploring the insides of altmetric.com”,El Professional De La Información, Vol. 23 No. 4,
pp. 359-366, doi: 10.3145/epi.2014.jul.03.
Roemer, R.C. and Borchardt, R. (2015), “Issues, controversies, and opportunities for altmetrics”,Library
Technology Reports, Vol. 51 No. 5, pp. 20-30.
Wouters, P. and Costas, R. (2012), Users, Narcissism and Control –Tracking the Impact of Scholarly
Publications in the 21st Century, SURF-foundation, Utrecht.
Corresponding author
Ann E. Williams can be contacted at: annwilliams@gsu.edu
For instructions on how to order reprints of this article, please visit our website:
www.emeraldgrouppublishing.com/licensing/reprints.htm
Or contact us for further details: permissions@emeraldinsight.com
317
Altmetrics: an
overview and
evaluation

Reproduced with permission of copyright
owner. Further reproduction prohibited
without permission.