Excellent research and societal impact in the 2021 UK REF

The UK REF can be considered the most extensive and influential evaluation system for research excellence and impact in the world. As a spectator sport, REF-watching is both entertaining and educational. Controversies over its impact on UK universities are numerous. The transparency of its design and process is both enviable and admirable. The rolling trough between breaking REF waves seems to be the inevitable liminoid moment when all the existential questions and concerns about the viability of the REF are aired and speculation intensifies. As we wallow between REF 2021 and REF 2029 the pattern is playing out again.

Calls have come to re-purpose the sizeable investment of time and resources that the REF absorbs and allocate research funding using a single metric. Experienced UK research policy participants wonder if it isn´t time to move on. Assessments find that while broadening the scope of the REF 2021 was a good thing, it was also more expensive. Seasoned REF whisperers push back on simplifying the REF when there is still work to do, and consider the next wave is going to be the best one yet. The research community itself seems to be somewhat on the fence.

Step back in time

At the beginning of OSIRIS, back in 2017, we wrote about on the relationship between excellent research and societal impact in the 2014 UK REF. All outputs to the REF are scored on a quality scale ranging from one to four ‘stars’, with four stars (4*) denoting ‘excellence’. We took a look at the relationship between the 4* components of evaluated scientific outputs and societal impacts for all unit submissions (typically departments). These 4* components of unit submissions were those assessed as “world-leading” (scientific outputs) and “outstanding in terms of reach and significance” (impacts) in their field.

For each unit submission the 2014 REF provided an assessment of the proportion of its submitted outputs and impact evidence that met the criteria for 4* standard. We plotted these proportions for all panels and sub-panels. What we found was that a great deal of dispersion existed across all the REF panels. It was apparent in all panels that a Unit submission can produce high levels of world-leading scientific outputs while not generating any outstanding impact. Alternatively, a Unit submission may generate very high levels of outstanding societal impact while producing little or no world-leading science. Most Unit submissions lay somewhere in between these two extremes.

In all four main panels, a positive correlation existed between the percentage of a Unit submissions’ scientific outputs and impacts that were assessed as 4*. This positive relationship was also evident in all subject sub-panels bar Clinical Medicine. However, these relationships were not particularly strong. We therefore suggested that the heterogeneity of the relationships might constitute the real strength of UK Universities.

Shocked (not)

Turning to the results of REF 2021, we again plotted panels and sub-panels. We would first note that there have been significant changes in the REF methodology in relation to the evaluation of impacts. No longer were submitting units required to provide impact cases that are representative of the research activity of their overall work. Rather, submissions should “select the strongest examples of impact that are underpinned by the submitted unit’s excellent research”. The number of impact cases provided depends on the size of the unit (by FTE staff). Impact cases that already features in the 2014 REF could be included again “where additional underpinning research has taken place”. For these main reasons we don’t make direct comparisons between REF 2014 and REF 2021 results here.

Figure 1 demonstrates the wide dispersion in correlations between 4* scientific outputs and associated impacts across all REF 2021 panels. It is evident in all panels that a unit submission can include high levels of 4* scientific outputs while not generating 4* societal impacts. Equally, a unit submission may generate very high levels of outstanding societal impact without producing much in the way of 4* scientific outputs. Positive correlations exist between the percentage of unit submissions’ scientific outputs and impacts that are assessed as 4* in all panels. There is little difference between Panels A, B and C, while for Panel D the correlation is a little weaker. None of these statistical relationships are particularly strong. Overall, the shape of these results is very much consistent with what we found for the 2014 REF.

At the level of sub-panels, there is some more variation evident. There is relative consistency among medicine and health fields (Panel A). More uneven results can be seen among STEM fields (Panel B), with particularly strong relationships between outputs and impacts in Engineering and ICTs. Within the social sciences (Panel C) most sub-panels show a similar positive correlation. The relevance of Education research is highlighted by the strongest correlation among all sub-panels. The two weakest relationships within social sciences are found in those sub-panels with a significant international focus, Anthropology and Development, and Politics and International Studies.

The greatest variation between sub-panels exists within the humanities (Panel D). A majority of fields are more or less on par with other sub-panels across the entire REF. However, in the Classics and Philosophy sub-panels there is a relatively weak correlation between outputs and impact. In the case of Area Studies this relationship is negative.

These results give us no reason to change our argument from 2017, that the great diversity of the UK research system may well constitute its greatest strength. As we could see back then, creating high-quality societal impacts from UK university research does not only come from producing the highest quality science. Rather, connections between high quality science and societal impacts again appear to be heterogenous and strength of the statistical connection between 4* outputs and impacts remains of similar strength.

Spinning around

So, is the repetition of the REF exercise a bit like riding an exercise bike, a lot of energetic pedaling without passing through any new scenery? That the concept of ‘underpinning research’ includes research outputs from the period 2000-2020 might suggest that there is indeed a significant element of ‘different but same’. But such a framework also creates space for medium and longer-term impact pathways to be recognised and rewarded - which is undoubtedly a positive.

Our results would suggest that nothing has changed in the relationship between excellent research and the significant societal impacts that are based on this research. We might question whether the intervening period between REF exercises is currently lengthy enough to allow new impacts to be felt, understood and appropriately documented. Perhaps one idea could be to vary the timing of assessments of different components of the REF, particularly as it again appears likely to expand in scope. From our perspective, perhaps expensive assessments of outputs and impacts are being conducted more frequently than is likely to lead to anything new being added to science and research policy intelligence, (whilst also absorbing substantial resources and person-hours).

On the other hand, REF 2029 promises to deliver a normative turn, with a stronger focus on the desirable qualities of the research environment and culture. There is little doubt that transforming who gets to do research and improving the conditions for this to happen is likely to be the ‘next big thing’ in terms of policy impact on research outcomes. In REF 2021, the importance of the research environment and research training components increased. Perhaps this ‘people and relationships’ focus should be brought fully to centre stage in 2029. This could promote a more compact and relevant agenda contributing to potentially profound institutional and organizational changes, while allowing research outputs and impacts a little more time to breathe and ripen.

While for many researchers and university administrators in the UK it may well be a case of ‘better the devil you know’, for dedicated REF-watchers it will likely be the changes and innovations that emerge in the run up to REF 2029 that are particularly instructive – and provide the most entertainment!

You can download and experiment with the data discussed in this post here.

About the authors

Richard Woolley is a researcher at Ingenio (CSIC-UPV), a science policy and innovation institute in Valencia, Spain. Nicolas Robinson-Garcia is a Ramón y Cajal Fellow and scientific director of the Unit for Computational Humanities and Social Sciences (U-CHASS) at the University of Granada, Spain.

This text is part of a series of blogs from the OSIRIS centre. We will publish a number of these blogs in the coming weeks leading up to the final conference that we organise together with R-QUEST (our sister centre focused on research quality) on November 8^th. To sign up for this conference, which will be held in Oslo but also streamed to a wider audience, click here.