Excellence dominates over impact in grant proposal assessments

In the early days of competitive research funding, grants tended to come with limited strings attached as the general belief was that autonomy for researchers would best serve to push the frontiers of science and that positive impacts for society would accrue as a result (Bush, 2020). As competitive funding is now increasingly also expected to contribute in a more directed fashion to solving the grand challenges facing humanity, it has become commonplace that research funders ask researchers to discuss the potential societal impact of their proposed research in their applications. However, this development is not uncontroversial. While scientific peers are generally seen as well adept at assessing the scientific impact of project proposals, their ability to judge societal impact is disputed.

Challenges in evaluating impact before it has happened

Assessing the societal impact of completed research is fraught with considerable challenges which tend to be aggravated or be different altogether when attempting to judge the potential impact of research not yet undertaken. The criteria and processes of these judgements are not standardized and are often ambiguous (Ma et al., 2020).

Reviewers vary in their interpretation and use of societal impact criteria and in their motivation, ability and willingness to carry out such assessments (Abma-Schouten et al., 2023; Bozeman & Boardman, 2009; Derrick & Samuel, 2016). Furthermore, there are indications that including societal impact considerations into the review process means short-term and commercially oriented research tend to be preferred (Ma et al., 2020).

Limited attention to impact in review panels

In OSIRIS, we have carried out extensive studies of review panels at work, with observations of almost 70 different panels and follow-up interviews with close to 90 reviewers. We find that the impact of incorporating societal impact criteria into the review process is often limited. If left to their own devices, reviewers tend to largely disregard societal impact criteria in their overall assessment of grant proposals (the underlying research is currently under review for a scientific journal).

Naturally, the degree to which reviewers attach importance to societal impact considerations will vary. As might be expected, disciplinary background is one source of variability, with applied scientists tending to value impact higher. Personal topic preferences also tend to exert influence, and reviewers often conflate their assessment of societal impact with their assessment of the importance of the topic as such.

Overall, we identify two main approaches among reviewers with respect to how they operationalize impact criteria. Most commonly, reviewers and panels express a linear understanding of impact creation which prioritizes plans for dissemination and communication. Less common is a more interaction-oriented approach, which considers plans for stakeholder involvement throughout the research process essential for achieving societal impact.

Excellence dominance

Despite these differences and nuances in how societal impact is understood and how important it is perceived to be, the dominant pattern is that reviewers attach primary importance to the assessment of scientific excellence. This is observed both in the amount of time spent on discussing different criteria and in how reviewers discuss why some proposals deserve to be funded. Assessing excellence is also a key area of expertise among the reviewers.

Funding criteria can be subject to processes of bargaining and gaming as panels discuss to come to an agreement about the final score (most panels required consensus). We observed that the impact score was more frequently changed or used as a bargaining chip. The reason is likely that impact is a newer and less standardized criterion. Furthermore, assessments of impact tended to follow assessments of excellence. For example, if a reviewer had a very positive judgement of a proposal’s excellence, the impact score was also often high without necessarily being based on a substantial assessment of impact as such.

This point is not normative. Panels are social contexts, and experts rarely agree completely about the assessment of proposals, which means that bargaining is necessary and useful. Putting more weight on an established criterion – like excellence – close to the reviewers’ own areas of expertise, might also be both understandable and rational.

Organizing for impact

Still, if funders of research want to give societal impact criteria similar emphasis as scientific excellence, as many of them state that they do, this needs to be reflected in how the review process is organized. In our data set, the most important aspect is to have dedicated societal impact criteria, which clearly indicate the impacts expected and with clearly assigned weights. Such an approach seems particularly suited to targeted calls for research, where applicants can be expected to have more substantial and clearer outlines of expected impacts.

The main alternative approach is to grant reviewers considerable discretion with respect to how societal impact is operationalized and weighted. We observe that this may lead to significant variations and a primary emphasis on scientific excellence. For calls that emphasise bottom-up, long-term fundamental research, such an outcome might be effective. But in these types of calls, perhaps societal impact is better promoted by other means than an explicit outline of it at the time of applying for funding?

This text is part of a series of blogs from the OSIRIS centre. We will publish a number of these blogs in the coming weeks leading up to the final conference that we organise together with R-QUEST (our sister centre focused on research quality) on November 8^th. To sign up for this conference, which will be held in Oslo but also streamed to a wider audience, click here.