What do the users of research want?

Is there a contradiction between quality and relevance? Do researchers emphasize originality and publication in prestigious journals, while users want easy-to-read and simple answers to practical questions? A new, large survey among users of research in public agencies indicates that users emphasize many of the same quality criteria as the researchers themselves.

The relationship between quality and relevance is a recurring theme in science policy, and the relationship is expressed today especially through terms excellence and impact, which are used in their English form even in Norway. Researchers often want some form of high quality research in a scholarly sense, characterized primarily as original contributions to a research area through publications in prestigious scientific journals. Many organizations and sectors outside of the research system are required to or aspire to use research. These actual or potential users want research to generate some sort of impact – an intended effect in the organization, the sector or society.

But do users and researchers have different perceptions of what makes research good and useful? To study how users view the research system and their own needs for research-based knowledge, we have carried out a large-scale survey with responses from almost 1,900 employees in ministries, directorates and other government agencies in Norway.

Prestige is important to users

We asked the employees about which characteristics of the research and the research environments that they saw as important for the relevance and applicability of the research. The results are shown in Figure 1.

Figure 1. Answer to the question "How important are the following aspects of the research for your assessment of the relevance / applicability of the research?"
Figure 1. Answer to the question "How important are the following aspects of the research for your assessment of the relevance / applicability of the research?"

The answers show that the users’ perceptions in many ways correspond to common quality criteria in research. Users are the most concerned with the researchers/research environments being reputable, but they also place great emphasis on publication in recognized journals and that the research is novel. The latter does not necessarily express that what is novel to a user will have scientific originality, but at least there seems to be no fundamental contradiction.

Only a minority of users emphasize previous knowledge of the research environment, that they have commissioned the research themselves, or that the results are published in Norwegian. At the same time, the users are concerned with the fact that the researchers should think about the application of the results: “That it (the research) clearly states what works”, and “that clear implications for policy formulation are described” are considered equally important as originality and even more important than publication in reputable journals.

Positive view of the research communities

Another set of questions was aimed at getting users’ assessments of the research in the users’ specific areas of work. The results are summarized in Figure 2.

Figure 2. Answer to the question "How much do you agree with the following statements about the research in the field you are working on?"
Figure 2. Answer to the question "How much do you agree with the following statements about the research in the field you are working on?"

The state-employed users largely agree that the researchers and the research communities are perceived as independent, that the communities are concerned with working with relevant issues, and that the research is of high quality. On the other hand, one third of the respondents also fully or partially agree that the research is characterized by a high degree of uncertainty, that it is inaccessible, and that the field has few competent academic environments/researchers.

There are several significant differences between user organizations. For example, respondents from the Ministry of Climate and Environment and the Ministry of Petroleum and Energy in particular expressed that their fields have competent research environments. At the opposite end of this scale we find the Ministry of Local Government and Modernisation, the Ministry of Culture, and the Directorate for Education and Training.

Do users who hold a doctoral degree emphasize other quality criteria?

Between five and ten percent of the respondents hold a doctoral degree and thus have experience from research and scientific publishing. However, their answers to the two sets of questions are very similar to the answers of the respondents without a doctoral degree, see Figure 3 and Figure 4. It can be added that many of the other users in the data material hold a master’s degree.

Figure 3. Differences between user employees with and without a PhD when asked what they think is important about research. Each point is the average value of the groups on a scale from 1 to 5, see top left.
Figure 3. Differences between user employees with and without a PhD when asked what they think is important about research. Each point is the average value of the groups on a scale from 1 to 5, see top left.

For the questions about important criteria in research, the respondents with a doctoral degree are less concerned with empirical data and language being Norwegian and that the implications are clearly presented, and they are significantly more concerned with novelty and publishing in recognized journals. Beside this, the differences are insignificant. Regarding the questions about research in their own field, the respondents with a doctoral degree are somewhat more positive about the independence and quality of the research communities, their preoccupation with and media visibility on relevant issues, and they to a lesser extent think that the research is uncertain.

Figure 4. Differences between employees with a PhD and those without when asking questions about how they assess the field of research. Here, each point is the average value of the groups on a scale from 1 to 5, see top left.
Figure 4. Differences between employees with a PhD and those without when asking questions about how they assess the field of research. Here, each point is the average value of the groups on a scale from 1 to 5, see top left.

In sum, the answers to these questions express that users in the public sector in many ways put emphasis on the same characteristics of research that the research communities themselves prioritize: originality, regular academic publication, independence, and in many cases an international orientation. From the researchers’ point of view, the most difficult aspect is probably the expectation that they should give clear recommendations for policy development and explain “what works”. In many situations, research might have unclear or several different implications, and it is also not certain that researchers always know specifically how policymaking takes place.

The survey also shows that there is essentially a trusting relationship between researchers and users in Norway, and – at least as a hypothesis – there might be less skepticism about the independence, relevance and quality of research than in other countries.

This blog was originally published as an article in Norwegian in the magazine Forskningpolitikk. You can access the full online version of it hereThe survey is part of the work of the OSIRIS center, which studies the societal effects of research. A questionnaire was administered to approximately 6000 government employees in 2019 and received responses from around 1900 people (28 percent). The survey contains information on several topics related to how government employees acquire, disseminate and use research-based knowledge in their work.

Tags: Use of research, research impact, policy impact, research in policy By Magnus Gulbrandsen, Erlend Simensen, Taran Thune
Published Oct. 26, 2020 6:50 PM - Last modified Oct. 26, 2020 6:50 PM
Add comment

Log in to comment

Not UiO or Feide account?
Create a WebID account to comment

illustrasjon

The OSIRIS blog

On the OSIRIS blog the members of the project team write about impact of research as our research on this topic progresses.

We aim for a collection of posts that represent preliminary and conceptual findings and ideas, discussions from meetings and seminars, shorter analyses of empirical data and brief summaries of the vast literature on impact. Some of the posts will be shared with the Impact Blog at the London School of Economics, the most comprehensive web page devoted to this topic and a great source of interesting ideas about many topics within science policy and science in practice.

The blog is also open for contributions from people outside of the OSIRIS team. Send us an email if you have a text that would fit into the blog.