In the process of assigning grant applications, the documents are subject to rule-based pre-processing steps, before they are prepared for machine reading. Then an algorithm identifies keywords and themes in the proposals. Based on this, the system suggests groups of experts for the evaluation. Finally, RCN case handlers decide whether to agree with the AI suggestion or move the application to a different expert panel.
This opens interesting questions about the fates of different proposals and the decision of whether humans or machines assign more “correctly”. Even in manual assignment processes, interdisciplinary applications can be challenging to place into a single expert panel. It will be interesting to see how the introduction of AI processing could change such challenges or create new ones.
Another task of the RCN is mapping research outputs in relation to its portfolio structure of thematic priorities. Funded grant proposals are processed with a topic modelling approach that predicts various properties of each project (academic disciplines, research themes, relevance to various sectors etc.). The predictions are reviewed by case officers who decides which tags to assign to the project.
RCN also produces standard bibliometric indicators for its funding portfolios. However, these remain limited to formalized knowledge in the form of research publications, and provide little knowledge of relevance to society outside of academia. RCN is currently investigating the potential of altemetrics to capture uptake of RCN-funded research in other outlets, for example in patents, news and blogs, Wikipedia articles, or policy documents. An algorithmic classification process assigns these materials into a taxonomy of media topics. This could also be used to identify gaps or missing signals in certain thematic areas. The classification process is based on Wikipedia data which seems to work very well for the types of documents examined here.
Nevertheless, questions emerge on the applicability of international data sources and pre-trained models in evaluating Norwegian research outputs. This also links to questions of excellence, open science, and other challenges in tracing impact and uptake of science, connecting to OSIRIS work in many interesting ways.