Exploring the use of research in policy making: insights from the literature and a pilot survey
In OSIRIS in 2018 we have tested a new method for mapping the use of research by practitioners and policymakers in the public sector. The results show that there is a large degree of diversity in how research is accessed and used. In general, informal practices like “asking a colleague” and “googling” are more frequent than formal ways of searching for research-based knowledge.
The questionnaire was created in an intensive two-day workshop in Manchester.
Limited knowledge base
Contrary to studies of the private sector or health care, there is no ongoing or standardised measurement system for research investments or impact of research in public policy organisations. For our pilot study, we collected existing literature and survey tools on utilisation of research in policymaking. In addition, we carried out interviews with senior policy officials in Norwegian ministries to develop and calibrate a survey to fit the specific context of policymaking.
Research impact might be a new term in research and innovation policy contexts, but the concept knowledge utilisation has had a long life in the literature, especially within medicine and healthcare. If we are ill, we assume that decisions made by medical doctors about how they treat us are based on up-to-date knowledge about the condition that affects us and possible responses to it. If health policymakers make decisions about issues such as treatment programs for cancer, nutritional programs for babies or the level of toxins allowed in food, we also expect that decisions are based on updated and validated scientific knowledge. This means that the research has been put to use – which is simply put what “research utilisation” refers to.
Within broader areas of policy, such as the passing of new laws, allocation of state budgets or implementation of a new program to support, for instance, innovation in industry, the link between policymaking and research is less straightforward. Most of us expect that the available evidence is at least taken into considerations when decisions are made on a specific issue. It is easier to dismiss decisions that are made on a whim, based on ideology alone, use selective references to research results or perspectives, or contradict important research findings. Research can help shed light on an issue and on ways forward, although cynics may perceive that policymakers often use science as a drunk man uses a lamppost – more for support than illumination.
We have set out to study in detail how individuals working in different areas of policy get access to scientific knowledge, how they diffuse it within their organisations, and the different ways that they use scientific knowledge in their work. In line with prior studies, we define research utilisation from the users’ perspective. We do not equate research utilisation and research impact, but see utilisation as a step on the journey towards impact. If research is not at all picked up, there will be no clear pathway to impact – although in the long run research results still might “creep” into policy or practice.
What factors influence research utilisation in policy?
We are particularly interested in factors that influence how policymakers’ use research. These can be individual level factors such as the age, education, position or prior work experience of policy officials, characteristics of the policy issues on which the informants work, and organisational level factors. Prior studies have found that the educational level and prior work experience from research organisations lead to increased research utilisation by policymakers. In other words, familiarity with research increases use of research-based knowledge. Having a leadership function or being a senior policy official also increases research utilisation.
Furthermore, if the policy organisation values scientific evidence highly, or if it works with policy issues that directly influences peoples’ lives and welfare, it is more likely to use research. Examples of such issues are climate, food, health, welfare and education. Policy areas that are connected to basic societal infrastructures, the economy or industry, to a lesser extent demand that policy decisions are based on scientific knowledge. Pre-survey interviews with senior policy officials in Norway confirm these expectations, and also indicate that the degree of controversy and public attention a policy area receives influence the likelihood of using research evidence.
Varied and informal use
Based on the interviews and a broad review of the literature, OSIRIS researchers from Oslo and Manchester developed a unique survey tool about knowledge utilisation. 277 employees in five Norwegian ministries and two state level agencies participated in the pilot survey during the spring of 2018.
The first results indicate that policy officials in Norway that often use research, match the characteristics found in earlier investigations: they are senior, they have a high education level and they have work experience from other sectors, particularly research sectors. Most respondents stated that they gain access to scientific knowledge in an informal manner, i.e. they search for information on the internet or they discuss with colleagues inside or outside their organisation. We find informal aspects also in the way knowledge is shared within the organisation, which happens mainly through inter-collegial networks.
Searching for research-based knowledge and distributing it internally is often guided by the need to solve specific problems. When asked about use – where we worked a lot to find good ways of asking the questions – the respondents highlighted that conceptual use (to provide new understanding of a policy issue) or instrumental use (to develop policies or programs) are the most frequent. Symbolic uses of knowledge (i.e. using the lamppost as support) is the least common in the pilot survey. For the respondents, it matters a lot that the research is strong on methods, that it is able to clearly articulate implications and “what works”, that it uses Norwegian data, and that the researchers are renowned. Citations and having prior dealings with the researchers matter little.
The sources of knowledge varies a lot. We asked about nine different sources, ranging from universities and colleges and a number of other research organisations to private firms, internal units and sector specific knowledge centres. All sources are used by at least some respondents. When looking at the participating ministries and agencies, the degree of frequent use varies a lot. Whereas three out of four respondents from one organisation used consultancy firms often or very often, none of the respondents from several other organisations said the same. The use of international organisations varies similarly.
These results are only indications. We do not have enough respondents to do systematic analyses of the differences between ministries, policy areas and so on. Our next step is therefore to do a full-scale survey with a larger number of respondents from more organisations, to allow us to explore such essential questions. This is planned for 2019, and we have started recruiting participating organisations in late 2018. In addition, we will do a test survey among policymakers in the UK, which can allow for some interesting cross-country comparisons.
Based on the pilot stage we have modified some of the questions, and we now have a powerful tool to map and compare the use of research-based knowledge in public agencies and ministries. If you represent an organisation interested in taking part, please contact Taran Thune or Magnus Gulbrandsen from the OSIRIS team.
- "Further Research is Needed!" May 24, 2019
- Exploring the use of research in policy making: insights from the literature and a pilot survey Nov. 30, 2018
- Does public R&D funding work? Sep. 14, 2018
On the OSIRIS blog the members of the project team write about impact of research as our research on this topic progresses.
We aim for a collection of posts that represent preliminary and conceptual findings and ideas, discussions from meetings and seminars, shorter analyses of empirical data and brief summaries of the vast literature on impact. Some of the posts will be shared with the Impact Blog at the London School of Economics, the most comprehensive web page devoted to this topic and a great source of interesting ideas about many topics within science policy and science in practice.
The blog is also open for contributions from people outside of the OSIRIS team. Send us an email if you have a text that would fit into the blog.