What is the economic meaning of ‘impact’?

One important side-effect (impact?) of the OSIRIS project has been to get us—social-science researchers of different stripes— to think about the ability of research to contribute to more than just academic outputs. It’s not necessarily a research finding, but it turns out that social scientists approach this question differently based on our disciplinary eyewear. Some initially think in terms of better public health outcomes, or improved cultural understanding, or greater positive societal change. Others naturally think in terms of improving public policy or, in the longer term, economic development.

These are of course all different ways to think about how research can contribute to improving societal well-being. Or social welfare if you’re among the more economically minded social scientists. But how can we zoom in and out between the narrower idea of welfare and the broader idea of well-being? Do we need disciplinary bifocals? Or can working together in the same project help us to focus concertedly on the societal impact of research, and how to improve it?

This short piece builds on the experience of working together, finding a common vocabulary, and discovering how complementary views beyond disciplinary stereotypes. We point out that some basic distinctions from economics can be useful when thinking about impact. The piece also emphasizes that the field is evolving as economically oriented research explores new methods and measures to improve public policy. A central point is that by improving public policy around financing research and development, research can continue to improve its societal impact.

Two social scientists walked into a project… How to understand the societal impact of publicly funded research? At the risk of caricature, the economist tends to focus on the investment (input) and on the ‘impact’ (output) elements, and about how to model and measure their relationship. Much less time will be allocated to the specifics about the ‘research’ process or the ‘societal’ context, let alone the interactions between the two—they are by and large happy to leave these ‘messy’ dimensions to their colleagues elsewhere in the faculty.  

This is of course a huge and, for some, totally unfair stereotype. But it is helpful to call out some important differences that, untreated, can create barriers between two otherwise complementary groups of social scientists. Let’s have a closer look.

The economist’s black box: A first problem this stereotype reveals involves how to think about the public investment in research and the output of that research in similar terms. This is actually a really complex issue. We can think of it this way: As an input, money invested in research is ‘homogeneous’; as research output, the stream of new knowledge that results is necessarily ‘heterogenous’. This mismatch is inconvenient for the economist, who therefore prefers to rebase inputs and outputs onto a common denominator, preferably money. Thus, money inputs lead to research output, that leads, after some lag, to an improved stock of aggregate input in the form of physical and human capital. These in time yield better outcomes for specific sectors of the economy, such as the business enterprise sector, or, in rarer circumstance, for the economy more widely.

If this sounds like a black box, it traditionally has been. It has also usually been accompanied by heroic assumptions to smooth over problems.  Aghion et al (2009) presents a nice critique of the field’s tendency to employ this “homogenizing device”. But the good news is that economists are increasingly looking around inside it, increasingly with other social scientists. In OSIRIS, for example. To think about how to improve public policy, for example around the initial aim of promoting research in the first place.

The head of the two-handed economist: The comic book economist presents findings in terms of ‘on the one hand…’ and ‘on the other…’. But the predilection of economists for distinctions can be useful. Let’s take the simple distinction between first and second order effects. The initial expectation is that research will yield findings that can be codified into publications or, in some specific cases, into patents. This direct, first order effect is the common conception of a ‘research output’.

In order to think about societal impact, a further step is necessary. Bornmann’s review (2013) distinguishes the more run-of-the-mill research output from broader ‘research outcomes’ that (may) emerge over time from the original investment. These are typically second order effects and can include a range of knowledge ‘products’, think for example of those engendered in improved components of a medicine, the business idea of a spinoff company or better solutions to municipal water supply.

Societal impact builds on public returns: In the move from output to outcome, products that stem from the investment can go on to have societal impact under given conditions. The distinction between private and public returns is useful here. The ‘private returns’ are typically those that accrue to the researcher in the case of 'results' such as increased standing in the field, say, or even patent royalties. It is the products of research, however, that can yield ‘public returns’. ‘Societal impact’ emerges traditionally through a widening range of ‘third-stream activities’ involving other parts of society. The idea is that public returns materialize if/when research leads to 'innovations' in a context that is relevant to societal wellbeing.

Value for money, yes, but what is your currency? These outcomes become ‘inputs’ into the economy. A crucial issue is then where to identify relevant contexts. In the review cited above, research may lead in different ways to social, cultural, environmental, and economic returns that stem from research results or actual outcomes. The implicit assumption is that the research results are taken up more widely to build directly on knowhow that has societal value.

Third-stream activities are inherently heterogeneous; they pursue various channels on their way to contribute to positive impacts. The question then is ‘positive for whom’? It depends: who is the user, what is the use-case, how useful is the research outcome in that context? Different ‘users’ will appreciate research outcomes – ‘innovations’— in terms of the ‘value’ they have in specific use cases. How much of this value spreads through to other societal actors will condition its societal impact. Experience suggests that this happens differently in different cases, defying the preference to homogenize outputs. This is one area where economists can learn from their colleagues.

What if there was no public investment in research? But to conclude, it is important to be aware that the public investment in research is not necessarily a given. Each year, Norway invests about 40 billion public NOK into research, including about 10 percent of the R&D carried out in the private sector. Society has long held an underspecified expectation that it will more or less directly contribute to societal well-being. But the implicit expectation that has been there is that the investment will yield better trained students, better solutions both to theoretical and to practical problems, improved cultural understanding, as well as better intermediate inputs to the economy.

In recent decades, society has tended to accept the large degree of uncertainty in research and innovation; about the immediate output vs the more long-term outcome, about its return vs the share of the return that is public, about the link with users in society etc. In part, as a result, it has left many of the choices to the science system, nudged on by a range of policy-driven allocations.

Things seem to be changing, however. Debates about public policy priorities have become more active in recent years. We find proponents and opponents of mission-oriented innovation policies in our newspapers, for example. There is the distinct possibility going forward that these discussions degenerate into arguments or special pleading, rather than deliberations that take into account the important distinctions noted here that the outcomes of this special form of public investment is hard to measure and may be even harder to convert into ‘value for money’ narratives.

Economists evolve: Social scientists are increasingly aware that the public investment is not necessarily a given. But there is good news. Beyond the headlines, there are serious attempts to improve public policy based on the improved state-of-the-art in empirical studies. The ‘new economics of industrial policy’ literature (Juhasz et al, 2023) exemplifies the drive to use better measurement, the careful application of causal inference, and a more 'nuanced and contextual understanding of the effects of industrial policy' in order to improve the impact of publicly funded research. For example, by reducing inefficiencies such as deadweight loss and rent seeking or by improving the overall quality of the science and innovation system.

To this end, one strand of the OSIRIS work studied the impact that public support of private sector R&D has on firm level innovation and performance. One result in this article in Research Policy,  is that R&D funding is most important to young firms and firms with little or no R&D experience prior to obtaining support. On the other hand, the study shows that public R&D support to incumbent R&D active firms does not have such an impact on firm outcomes. This is an example of work aimed to inform public policy and to improve the impact that publicly financed research can have on society as a whole.

_______

This text is part of a series of blogs from the OSIRIS centre. We will publish a number of these blogs in the coming weeks leading up to the final conference that we organise together with R-QUEST (our sister centre focused on research quality) on November 8^th. To sign up for this conference, which will be held in Oslo but also streamed to a wider audience, click here.