Evaluating Research Impact from the User’s Perspective

Research investments, outputs and impacts

The analysis of research impact and its evaluation has typically been based, explicitly or implicitly, on models that take as point of departure instances of research activity or their outputs. We have at our disposal multiple tools, methods, and techniques to assess and trace the impact of a specific research investment or set of investments - or even the total of R&D investment at the national level. From this perspective, research users can often be seen as passive beneficiaries of outputs attributable to the research investment, whether in the shape of new products, services, or organisational changes.

Research users are also frequently seen as the recipients of research outputs who will then develop them further, into new outcomes and innovations. And, increasingly, research users are also viewed as, and incentivised to become, active partners in the research process. Regardless, the focal point of evaluation remains the research activity or its results.

The user perspective

Yet, to contribute to society, research results need to be used. Research users are thus critical to the processes of creating value from research (research valorisation). When the focus moves to the user, we find that the way users relate to research is seldom by reference to a single investment or outcome of research. Instead, their application of knowledge will typically draw from different research sources. A new or improved product will be based on a wide set of technologies embodied in its components and production processes. A policy decision will be based, often implicitly or even inadvertently, on a wide body of theoretical assumptions and research results.

Assessing impact by tracing from the research activity or its outputs presents a well-bounded initial point of departure. Alternatively, starting with a specific application implies the study of multiple and quite possibly scattered sets of potentially influential lines of research. Focusing on the role of users in bringing together these diverse strands thus constitutes a very different starting point for analysis.

This different starting point introduces new methodological challenges. Relevant interaction networks will appear more diverse and distributed. There will be a broad set of ‘productive interactions’ between users and researchers. A detailed and comprehensive analysis of these interactions may, in some contexts, be almost impossible due to the sheer variety of direct and indirect engagements that would need to be considered. Despite such difficulties, what will emerge by starting from the user perspective is a very different depiction of research impact processes.

Learning to see research like a user

We propose a set of principles to tackle this analytical challenge. These principles provide a general methodological framework for the analysis of knowledge valorisation processes from the user side:

1. Consider research user identities as actively and relationally defined. The role of the user in traditional, research-led models of impact is defined in relation to the research outputs, and the effort to generate value from them. From a user perspective, however, the role of the user is defined by its relationship with many actors involved in knowledge generation. This calls for a detailed analysis of where users are positioned in relation to research and researchers, their proximities in terms of geographical and temporal space, values, objectives, and ideologies. It is particularly important to know how users are positioned in relation to the processes by which problems are identified and defined. Problematisation from the user side is often narrowly defined as solution oriented. But this assumes that problems are well known, and that the users just focus on how to overcome them. Yet, users will often try to understand and define the problems, as well as assess the value that could flow from research, both as narrow solutions to well-identified problems but also as initially unforeseen opportunities for valorisation that the research can open up. A comprehensive model tracing the process of problematisation from the perspective of research users would include cognitive, technical, social and institutional dimensions.
2. Assume that the role of a research user is not fixed but changes over time. There are no a priori ‘research users’, entities that simply exist ‘out there’ in society waiting for a research result to drop off the end of the knowledge production line. Insisting on a separation between science and society or between researchers and communicators, when it comes to understanding how value is generated from research will, at best, generate an incomplete view of how these processes unfold. Research value is generated by multiple actors whose role may change; which actors are using research knowledge and for what purpose will vary across time. For instance, users may become brokers when they try to get other groups interested in the research results. These roles may be taken up simultaneously. They may also have multiple simultaneous roles. From this perspective, it then becomes important to understand the conditions that allow a variety of actors to shift among different roles to extend and diversify valorisation processes and increase their flexibility.
3. Reveal how research users pursue objectives relevant to their main field of operation by contributing to the generation of value from research. From a user perspective research impact assessment must include how users conceptualise and realise the value they can generate from research in their own area of operation. Research valorisation processes, whether ultimately ‘successful’ or not, can contribute to the accumulation of capitals in the users’ field of operation, thereby improving their position in that field. What types of outcomes that the user considers to be a success must be included in the analysis of research impact from the user side.
4. Situate research users as part of valorisation networks with distributed agency. Research users are involved in their own innovation, product development, service provision and other types of networks. The collective work of thes networks is necessary to generate value from research, but not all actors in the network are interested in the same features and characteristics of a potential innovation. For instance, not all actors involved in the valorisation process for a new drug are equally concerned with its efficacy, and its effects on public health. An actor may use research valorisation to take an enhanced position as a service provider within these networks. Understanding these occurrences becomes a further important reference point for the analysis of research impact from a user perspective. It needs to reveal the actors or networks mobilised by research use and the different ways in which they engage in research valorisation in order to foster their own interests.
5. Seek to identify final beneficiaries of research value that shift identities to become research users engaged in valorisation processes. Here, we understand by beneficiaries all those who benefit from research without actively contributing or working towards deriving value from it. They can also shift roles and become research users engaged in valorisation processes. For instance, consumers of mass-produced items sometimes engage in their adaptation and in this role they may draw on research results (witness for instance hacker communities). Therefore, when assessing impact from the user perspective it is also necessary to analyse when, how, and why identities shift from being detached from the “knowledge-value collective” to become an active part of it.

These principles can be operationalised by applying them to the analysis of how users engage with (1) research and innovation (strategy), (2) problem definition and operationalisation (problematizations), (3) in research valorisation processes (translation) and, (4) mobilise additional partners, capabilities and other resources to realise value in their fields of operation (capitalisation).

Overall, these principles and suggestions are an attempt to propose avenues to expand research impact assessment agendas to consider user roles, expectations, and dynamics more fully. They may direct attention to aspects of research impact processes that have not yet received the attention they deserve. For instance, the ‘fitness’ of different knowledge agents to cycle through the research user role is not well understood. A perspective emphasizing such user perspectives could provide a complementary way of seeing that enriches current theory-based assessments of the effects of specific research investments, organizations, programmes, and activities.

You can read more about research impact from the user perspective here.

Professor Jordi Molas-Gallart and Richard Woolley both work at INGENIO (CSIC-UPV) in Valencia, Spain.

This text is part of a series of blogs from the OSIRIS centre. We will publish a number of these blogs in the coming weeks leading up to the final conference that we organise together with R-QUEST (our sister centre focused on research quality) on November 8^th. To sign up for this conference, which will be held in Oslo but also streamed to a wider audience, click here.