Norwegian version of this page

Jan Ole Hesselberg

PhD candidate

Email janohes@student.sv.uio.no

Username

Download business card

The PhD Project

I Norway, several billion Euros are granted through application processes. However, the reviewers often disagree in their scoring of the proposals.

Why are the agreement levels low? And what can be done about it?

That is the topic of my project.

Funding and collaboration

The project is funded by the Dam Foundation (Stiftelsen Dam).

The Norwegian Cancer Society, The Kavli Trust and The Norwegian Research Council contribute with review data from their funding programs.

Academic interests

Judgement and decision making
Peer review
Open science

Background

I am a psychologist with a specialisation in clinical adult psychology.

I am the chief program officer at the Dam Foundation and the program manager of the Kavli Trust Programme on Health Research. I am also a member of The Norwegian National Committees for Research Ethics and I host a podcast called "Psykologlunsj".

Back in the days I worked on two popular science shows on Norwegian TV. "Folkeopplysningen" and "Typisk deg med Petter Schjerven".

For more info, check out my LinkedIn profile.

Research team

Supervisors

Knut Inge Fostervold, Associate Professor, Department of Psychology, University of Oslo
Pål Ulleberg, Associate Professor, Department of Psychology, University of Oslo
Ida Svege, Head of program development, Stiftelsen Dam

Hesselberg, Jan Ole; Ulleberg, Pål; Sørensen, Øystein; Fostervold, Knut Inge; Ingvaldsen, Sigrid Hegna & Svege, Ida (2026). Why do reviewers disagree? Evidence from four funders and 134,000 reviews. PLOS ONE. 21(9), p. 1–21. doi: 10.1371/journal.pone.0356108. Full text in Research Archive
Nilsonne, Gustav; Wieschowski, Susanne; DeVito, Nicholas J.; Salholz-Hillel, Maia; Ahnström, Love & Brückner, Till Daniel [Show all 41 contributors for this article] (2025). Results reporting for clinical trials led by medical universities and university hospitals in the nordic countries was often missing or delayed. Journal of Clinical Epidemiology. ISSN 0895-4356. 181. doi: 10.1016/j.jclinepi.2025.111710. Full text in Research Archive
Hesselberg, Jan Ole; Ulleberg, Pål; Sørensen, Øystein; Fostervold, Knut Inge; Ingvaldsen, Sigrid Hegna & Svege, Ida (2025). Registered report protocol: Factors associated with inter-rater agreement in grant peer review. PLOS ONE. 20(5). doi: 10.1371/journal.pone.0322696. Full text in Research Archive
Seeber, Marco; Svege, Ida & Hesselberg, Jan Ole (2024). Improving the efficiency of research proposals evaluation: A two-stage procedure. Research Evaluation. ISSN 0958-2029. p. 1–11. doi: 10.1093/reseval/rvae020. Full text in Research Archive Show summary
An important share of research funding is allocated via competitive programs, which entail considerable direct and indirect costs, such as to develop and evaluate the proposals. The goal of this article is to explore whether adopting a two-stage evaluation procedure could improve the efficiency of the process. For this purpose, we study the evaluation system designed by the Foundation Dam (Stiftelsen Dam), one of the largest foundations in Norway supporting health research. In 2020, Foundation Dam adopted a new evaluation procedure consisting in a short proposal for a first stage of selection and, for those selected, a second-stage evaluation of a long proposal. We explore whether such a procedure reduces the evaluation costs and how the evaluation procedures compare in terms of reliability. Survey responses from 94 of the 594 applicants in the one-stage procedure (2018–19) and all the 668 applicants in the two-stage procedure (2020–21) show that the two-stage procedure reduced the average time that applicants spent in drafting the proposal(s) by 38%. According to the reviewers’ estimate, the two-stage procedure also reduced by 28% the time they spent on average to evaluate an applicant’s proposal(s). The analysis of the evaluation data of 594 long proposals in the one-stage procedure and 668 short proposals in the two-stage procedure shows that reviewers’ scores of short proposals display greater reliability and agreement than the reviewers’ scores of long proposals in the old one-stage procedure. Our findings suggest that a two-stage procedure can substantially improve the efficiency of grant writing and review, without harming the reliability of the evaluation.
Hesselberg, Jan Ole; Dalsbø, Therese Kristine; Strømme, Hilde; Svege, Ida & Fretheim, Atle (2023). Reviewer training for improving grant and journal peer review. Cochrane Database of Systematic Reviews. doi: 10.1002/14651858.MR000056.pub2. Full text in Research Archive Show summary
Background Funders and scientific journals use peer review to decide which projects to fund or articles to publish. Reviewer training is an intervention to improve the quality of peer review. However, studies on the effects of such training yield inconsistent results, and there are no up‐to‐date systematic reviews addressing this question. Objectives To evaluate the effect of peer reviewer training on the quality of grant and journal peer review. Search methods We used standard, extensive Cochrane search methods. The latest search date was 27 April 2022. Selection criteria We included randomized controlled trials (RCTs; including cluster‐RCTs) that evaluated peer review with training interventions versus usual processes, no training interventions, or other interventions to improve the quality of peer review. Data collection and analysis We used standard Cochrane methods. Our primary outcomes were 1. completeness of reporting and 2. peer review detection of errors. Our secondary outcomes were 1. bibliometric scores, 2. stakeholders' assessment of peer review quality, 3. inter‐reviewer agreement, 4. process‐centred outcomes, 5. peer reviewer satisfaction, and 6. completion rate and speed of funded projects. We used the first version of the Cochrane risk of bias tool to assess the risk of bias, and we used GRADE to assess the certainty of evidence. Main results We included 10 RCTs with a total of 1213 units of analysis. The unit of analysis was the individual reviewer in seven studies (722 reviewers in total), and the reviewed manuscript in three studies (491 manuscripts in total). In eight RCTs, participants were journal peer reviewers. In two studies, the participants were grant peer reviewers. The training interventions can be broadly divided into dialogue‐based interventions (interactive workshop, face‐to‐face training, mentoring) and one‐way communication (written information, video course, checklist, written feedback). Most studies were small. We found moderate‐certainty evidence that emails reminding peer reviewers to check items of reporting checklists, compared with standard journal practice, have little or no effect on the completeness of reporting, measured as the proportion of items (from 0.00 to 1.00) that were adequately reported (mean difference (MD) 0.02, 95% confidence interval (CI) −0.02 to 0.06; 2 RCTs, 421 manuscripts). There was low‐certainty evidence that reviewer training, compared with standard journal practice, slightly improves peer reviewer ability to detect errors (MD 0.55, 95% CI 0.20 to 0.90; 1 RCT, 418 reviewers). We found low‐certainty evidence that reviewer training, compared with standard journal practice, has little or no effect on stakeholders' assessment of review quality in journal peer review (standardized mean difference (SMD) 0.13 standard deviations (SDs), 95% CI −0.07 to 0.33; 1 RCT, 418 reviewers), or change in stakeholders' assessment of review quality in journal peer review (SMD −0.15 SDs, 95% CI −0.39 to 0.10; 5 RCTs, 258 reviewers). We found very low‐certainty evidence that a video course, compared with no video course, has little or no effect on inter‐reviewer agreement in grant peer review (MD 0.14 points, 95% CI −0.07 to 0.35; 1 RCT, 75 reviewers). There was low‐certainty evidence that structured individual feedback on scoring, compared with general information on scoring, has little or no effect on the change in inter‐reviewer agreement in grant peer review (MD 0.18 points, 95% CI −0.14 to 0.50; 1 RCT, 41 reviewers, low‐certainty evidence).
Hesselberg, Jan Ole; Fostervold, Knut Inge; Ulleberg, Pål & Svege, Ida (2021). Individual versus general structured feedback to improve agreement in grant peer review: a randomized controlled trial. Research Integrity and Peer Review. 6(12). doi: 10.1186/s41073-021-00115-5. Full text in Research Archive Show summary
Background Vast sums are distributed based on grant peer review, but studies show that interrater reliability is often low. In this study, we tested the effect of receiving two short individual feedback reports compared to one short general feedback report on the agreement between reviewers. Methods A total of 42 reviewers at the Norwegian Foundation Dam were randomly assigned to receive either a general feedback report or an individual feedback report. The general feedback group received one report before the start of the reviews that contained general information about the previous call in which the reviewers participated. In the individual feedback group, the reviewers received two reports, one before the review period (based on the previous call) and one during the period (based on the current call). In the individual feedback group, the reviewers were presented with detailed information on their scoring compared with the review committee as a whole, both before and during the review period. The main outcomes were the proportion of agreement in the eligibility assessment and the average difference in scores between pairs of reviewers assessing the same proposal. The outcomes were measured in 2017 and after the feedback was provided in 2018. Results A total of 2398 paired reviews were included in the analysis. There was a significant difference between the two groups in the proportion of absolute agreement on whether the proposal was eligible for the funding programme, with the general feedback group demonstrating a higher rate of agreement. There was no difference between the two groups in terms of the average score difference. However, the agreement regarding the proposal score remained critically low for both groups. Conclusions We did not observe changes in proposal score agreement between 2017 and 2018 in reviewers receiving different feedback. The low levels of agreement remain a major concern in grant peer review, and research to identify contributing factors as well as the development and testing of interventions to increase agreement rates are still needed. Trial registration The study was preregistered at OSF.io/n4fq3.
Hesselberg, Jan Ole; Dalsbø, Therese Kristine; Strømme, Hilde; Svege, Ida & Fretheim, Atle (2020). Reviewer training for improving grant and journal peer review. (Protokoll). Cochrane Database of Systematic Reviews. doi: 10.1002/14651858.MR000056. Full text in Research Archive
Teigen, Karl Halvor; Andersen, Bjørn Sørskot; Alnes, Sigurd Lerkerød & Hesselberg, Jan Ole (2019). Entirely possible overruns: How people think and talk about probabilistic cost estimates. International Journal of Managing Projects in Business. ISSN 1753-8378. 13(2), p. 293–311. doi: 10.1108/IJMPB-06-2018-0114. Full text in Research Archive Show summary
Purpose The purpose of this paper is to examine people’s understanding and evaluation of uncertainty intervals produced by experts as part of a quality assurance procedure of large public projects. Design/methodology/approach Three samples of educated participants (employees in a large construction company, students attending courses in project management and judgment and decision making, and judges of district and appeal courts) answered questionnaires about cost estimates of a highway construction project, presented as a probability distribution. Findings The studies demonstrated additivity neglect of probabilities that are graphically displayed. People’s evaluations of the accuracy of interval estimates revealed a boundary (a “cliff”) effect, with a sharp drop in accuracy ratings for outcomes above an arbitrary maximum. Several common verbal phrases (what “can” happen, is “entirely possible” and “not surprising”) which might seem to indicate expected outcomes were regularly used to describe unlikely values near or at the top of the distribution (an extremity effect). Research limitations/implications All judgments concerned a single case and were made by participants who were not stakeholders in this specific project. Further studies should compare judgments aided by a graph with conditions where the graph is changed or absent. Practical implications Experts and project managers cannot assume that readers of cost estimates understand a well-defined uncertainty interval as intended. They should also be aware of effects created by describing uncertain estimates in words. Originality/value The studies show how inconsistencies in judgment affect the understanding and evaluation of uncertainty intervals by well-informed and educated samples tested in a maximally transparent situation. Readers of cost estimates seem to believe that precise estimates are feasible and yet that costs are usually underestimated.

View all works in NVA

Holter, Katrine; Hesselberg, Jan Ole & Spurkland, Kai (2025). #74: Dømt i media, frikjent av Høyesterett: Ny grense for politivold? Full text in Research Archive Show summary
Podcast Episode · Rett og Slett · 10/15/2025 · 1h 14m
Nordby, Kristin Jonassen; Mamelund, Svenn-Erik & Hesselberg, Jan Ole (2025). Ny studie: Tilfeldigheter og skjevheter påvirker hvem som får forskningsstøtte. [Internet]. Khrono. doi: https:/www.khrono.no/ny-studie-tilfeldigheter-og-skjevheter-pavirker-hvem-som-far-forskningsstotte/994505. Full text in Research Archive Show summary
Stipendiat Jan-Ole Hesselberg kritiserer Forskningsrådet og andre som deler ut støtte til forskning, for å ta for lett på likebehandling av søknader: — Hvem som vurderer søknaden har nesten like mye å si som kvaliteten på søknaden.
Røislien, Jo; Frøslie, Kathrine Frey; Stavseth, Marianne Riksheim; Langaas, Mette; Kvaløy, Jan Terje & Hesselberg, Jan Ole (2024). Nakne Tall (6:9) "Er du helt sikker?". [Internet]. Podkast NettOp UiS. Full text in Research Archive Show summary
Verden er full av usikkerhet, og svaret på enhver statistisk analyse av tall vil derfor også ha usikkerhet bakt inn i seg. Jo Røislien snakker med psykolog og forskningsfinansiør Jan-Ole Hesselberg om pizza og powerposing og hvor galt det kan gå når man fisker rundt i tallene på leting etter noe spennende. Ekspertpanelet forsøker å forklare statistikkens kanskje mest brukte og misbrukte tall: p-verdien.
Røislien, Jo & Hesselberg, Jan Ole (2023). Hvor ofte er aldri? Tidsskrift for Den norske legeforening (Tidsskriftet). ISSN 0029-2001. 143(18), p. 1–3. doi: 10.4045/tidsskr.23.0731. Full text in Research Archive Show summary
Vi bruker ord for å beskrive usikkerhet og sannsynlighet hele tiden. Samtidig er vi ofte svært uenige om hva disse ordene faktisk betyr.
Meyer, Jonas Hansen; Jordal, Preben; Økland, Ingunn; Søbye, Espen; Kvam, Anna Serafima Svendsen & Hesselberg, Jan Ole [Show all 8 contributors for this article] (2022). "Allerede før boken er åpnet" – om (in)habilitet i litteraturkritikken. [Journal]. Prosa - tidsskrift for skribenter. Full text in Research Archive
Hesselberg, Jan Ole (2025). Evaluation of a New Case-Processing Model in NEM. Jan-Ole Hesselberg. doi: 10.5281/ZENODO.22029443. Full text in Research Archive Show summary
In the autumn of 2023, the Norwegian National Committee for Medical and Health Research Ethics (NEM) decided to trial a new model for processing appeal cases: no rapporteur, mandatory tentative assessments from all members ahead of the meetings, and a structured discussion in which everyone first presents arguments for and against possible outcomes before open discussion and the final decision. The report describes the background for the changes, the new model, and the evaluation of the trial from October 2023 to October 2025: the members', chair's and secretariat's experiences, time use, engagement, decision types, dissents, changes from tentative to final assessments, agreement before and after discussion and self-assessed certainty. The committee recommends that the main features of the model be continued, with some adjustments. This is the English edition (translation) of the Norwegian report.
Hesselberg, Jan Ole (2025). Evaluering av ny modell for saksbehandling i NEM. Jan-Ole Hesselberg. doi: 10.5281/ZENODO.22029441. Full text in Research Archive Show summary
Høsten 2023 besluttet Nasjonal forskningsetisk komité for medisin og helsefag (NEM) å prøve ut en ny modell for behandling av klagesaker: ingen saksordfører, krav om tentative vurderinger fra alle medlemmer i forkant av møtene, og en strukturert diskusjon der alle først legger frem argumenter for og imot mulige utfall før åpen diskusjon og endelig beslutning. Rapporten beskriver bakgrunnen for endringene, den nye modellen og evalueringen av utprøvingen fra oktober 2023 til oktober 2025: medlemmenes, leders og sekretariatets erfaringer, tidsbruk, engasjement, vedtakstyper, dissenser, endringer fra tentativ til endelig vurdering, enighet før og etter diskusjon og egenvurdert sikkerhet. Komitéen anbefaler at hovedtrekkene i modellen videreføres, med enkelte justeringer.
Hesselberg, Jan Ole (2025). Improved Application Processing: The research, analyses and arguments that shape the basis for how Foundation Dam makes decisions. Stiftelsen Dam. ISSN 9788269194784. doi: 10.5281/ZENODO.21858496. Full text in Research Archive Show summary
Second edition (Report 2025:3) of Foundation Dam's report on grant application processing. The report presents the research, analyses and arguments behind the foundation's decision processes: continuous and two-stage application processing, reliability and validity of expert assessments (noise and bias), applicant and administrative burden, the organisation of expert panels and panel meetings, rating scales and decision rules, and the measures the foundation has introduced and evaluated since the first edition in 2020. The PDF edition is the version of record; a web edition is available at johes.no/en/reports/improved-application-processing. English edition of the Norwegian report Bedre søknadsbehandling.
Hesselberg, Jan Ole (2025). Bedre søknadsbehandling: Forskningen, analysene og argumentene som ligger til grunn for hvordan Stiftelsen Dam tar beslutninger. Stiftelsen Dam. ISSN 9788269194753. doi: 10.5281/ZENODO.21859060. Full text in Research Archive Show summary
Andre utgave (Rapport 2025:3) av Stiftelsen Dams rapport «Bedre søknadsbehandling». Rapporten beskriver forskningen, analysene og argumentene bak stiftelsens søknadsbehandling: løpende og to-trinns prosesser, reliabilitet og validitet i ekspertvurderinger (støy og bias), ressursbruk hos søkere og administrasjon, organiseringen av ekspertutvalg og gruppemøter, karakterskala og beslutningsregler, samt tiltakene stiftelsen har innført og evaluert siden første utgave i 2020.

View all works in NVA

Published Mar. 12, 2020 9:12 AM - Last modified Mar. 11, 2024 11:29 AM