Discussing indicators in research funding: What role do altmetrics play?

Altmetrics track down and count the mentions of scholarly outputs in social media, news sites, policy papers, and social bookmarking sites. To what extent are they used and valued to measure impact in research funding? This post was originally published by the Europe of Knowledge blog.

Image of computer screen with coding

Photo by Jacob Miller on Unsplash

Altmetrics track down and count the mentions of scholarly outputs in social media, news sites, policy papers, and social bookmarking sites. Then altmetrics data providers aggregate the number of mentions. This allows an observation of how many times research has been viewed, discussed, followed, shared, and downloaded.

By following this line of thought, one might relate these mentions to impact or attention in the wider public or the society outside of the scientific community. As such, everyone with an internet connection would be able to engage with scholarly outputs online, even if only a fraction of the overall number of users do so. Nevertheless, it is important to note these mentions do not correlate with the quality of a scholarly output, they mostly visualise a community of attention, that is internet users that engage in some or way or the other with a scholarly output, such as a journal article.

Following this development, altmetrics have reached the highest levels in European policy debates, and have been discussed, for instance, during the Open Science Mutual Learning Exercise (MLE) by the Horizon 2020 Policy Support Facility. MLEs are carried out under the Joint Research Centre Research and Innovation Observatory (RIO), and are aimed at providing the best practice examples from European Union (EU) Member States, and Associated Countries (European Commission, 2017b). Further evidence can be found in EU high-level expert groups that advise the European Commission, among others, on science, research, and innovation. From 2016 until 2017, altmetrics have been discussed in several reports of these high-level advisory bodies (European Commission, 2017a). At any rate, altmetrics, or alternative metrics, are gaining momentum in higher education (Holmberg, 2016), and altmetrics is an innovation with potential for further development (Bornmann, 2014; CWTS, 2017; Holmberg, 2016; Liu & Adie, 2013; Piwowar, 2013; Priem, Taraborelli, Groth, & Neylon, 2010; Robinson-García, Torres-Salinas, Zahedi, & Costas, 2014; Thelwall, Haustein, Larivière, Sugimoto, & Bornmann, 2013). This post is based on my master’s thesis (Fraumann, 2017) that explores the usage of altmetrics with a focus on research funding.

Altmetrics in research funding: rarely used, and unknown

For this study, representatives of a research funding organisation, and policymakers were first interviewed. Second, reviewers of a research funding organisation and researchers registered with an institutional altmetrics system were invited to take an online survey. Overall, the survey respondents and interviewees were unaware of the usage of altmetrics. The data also suggests a few of respondents are well-aware of the debates on altmetrics. If one closely follows the international debates on the usage of altmetrics, it might come as a surprise that the concept is so widely unused in this sample. It was expected that more respondents would be aware on the usage of altmetrics. In particular, if altmetrics are discussed in high-level policy debates in EU research policy, researchers need to be made aware of it, because this might also affect their academic career to some extent.

An outlook on altmetrics in research funding

As discussed before, altmetrics seems to be on the rise in policy papers and further international initiatives, such as at the level of EU policy. In turn, the findings that could be drawn from this sample of stakeholders suggest that altmetrics are not yet widely spread. In fact, they were unknown to the vast majority of the study participants. Furthermore, findings from the interviews also showed that different organisational types, academic disciplines, and further categories have to be treated differently.

As proven in several technical studies, altmetrics are not yet ready for routine use in research evaluations, and several challenges need to be addressed (Erdt, Nagarajan, Sin, & Theng, 2016). Nevertheless, through altmetrics, it is possible to make a certain impact on the society visible or to visualise attention. How this impact is interpreted and set into context is essential.

Additionally, it was suggested by some interviewees that altmetrics might play a larger role in reporting on funded research rather than demonstrating impact in research funding applications. Criticisms were put forward by some respondents on altmetrics. Further, altmetrics should only be seen as a complementary measurement compared to citation counts and, especially, peer review. For instance, the impact of sharing a research data set can be made visible in a timely manner compared to citation counts of a journal article. The context of altmetrics data and aggregated scores needs to be analysed, as suggested by several scholars.

As previously mentioned, the study findings for this sample of stakeholders in research funding indicate that altmetrics are mostly unknown. This needs to be considered if and when the usage of altmetrics is proposed by policymakers.  

References

Bornmann, L. (2014). Do altmetrics point to the broader impact of research? An overview of benefits and disadvantages of altmetrics. Journal of Informetrics, 8(4), 895–903. http://doi.org/10.1016/j.joi.2014.09.005

CWTS. (2017). CWTS Research Line in Altmetrics. Retrieved June 5, 2017, from https://www.cwts.nl/research/working-groups/societal-impact-of-research/altmetrics

Erdt, M., Nagarajan, A., Sin, S. J., & Theng, Y. (2016). Altmetrics: an analysis of the state-of-the-art in measuring research impact on social media. Scientometrics, 109(2), 1117–1166. http://doi.org/10.1007/s11192-016-2077-0

European Commission. (2017a). Europe’s future – open innovation, open science, open to the world: reflections of the Research, Innovation and Science Policy Experts (RISE) High Level Group. Brussels.

European Commission. (2017b). Mutual Learning Exercise on Open Science: Altmetrics and Rewards under the Horizon 2020 Policy Support Facility (PSF). Second Workshop on “How to use Altmetrics in a context of Open Science.” Retrieved from https://rio.jrc.ec.europa.eu/sites/default/files/Agenda MLE Open Science_Meeting 31 May 2017_Helsinki.pdf

Holmberg, K. (2016). Altmetrics for information professionals: Past, present and future. Waltham, MA: Chandos Publishing.

Liu, J., & Adie, E. (2013). New perspectives on article-level metrics: Developing ways to assess research uptake and impact online. Insights: The UKSG Journal, 26(2), 153–158. http://doi.org/10.1629/2048-7754.79

Piwowar, H. (2013). Altmetrics: Value all research products. Nature, 493(7431), 159. http://doi.org/10.1038/493159a

Priem, J., Taraborelli, D., Groth, P., & Neylon, C. (2010). altmetrics: a manifesto. Retrieved from http://www.altmetrics.org/manifesto

Robinson-García, N., Torres-Salinas, D., Zahedi, Z., & Costas, R. (2014). New data, new possibilities: exploring the insides of Altmetric.com. El Profesional de La Informacion, 23(4), 359–366. http://doi.org/10.3145/epi.2014.jul.03

Thelwall, M., Haustein, S., Larivière, V., Sugimoto, C. R., & Bornmann, L. (2013). Do Altmetrics Work? Twitter and Ten Other Social Web Services. PLoS ONE, 8(5), e64841. http://doi.org/10.1371/journal.pone.0064841

About the author

Grischa Fraumann is a graduate of the Master in Research and Innovation in Higher Education (MARIHE) at University of Tampere (Finland) and Danube University Krems (Austria). This blog post is based on his master’s thesis: ‘Valuation of altmetrics in research funding’.

Tags: Evaluation, societal impact, altmetrics, research funding By Grischa Fraumann
Published July 3, 2018 2:13 PM - Last modified July 29, 2022 10:59 AM
Add comment

Log in to comment

Not UiO or Feide account?
Create a WebID account to comment

illustrasjon

The OSIRIS blog

On the OSIRIS blog the members of the project team write about impact of research as our research on this topic progresses.

We aim for a collection of posts that represent preliminary and conceptual findings and ideas, discussions from meetings and seminars, shorter analyses of empirical data and brief summaries of the vast literature on impact. Some of the posts will be shared with the Impact Blog at the London School of Economics, the most comprehensive web page devoted to this topic and a great source of interesting ideas about many topics within science policy and science in practice.

The blog is also open for contributions from people outside of the OSIRIS team. Send us an email if you have a text that would fit into the blog.