Background
A group of scientists within the Fraser, Coyote-Maestas, and Pinney labs have begun a journal club centered around issues of diversity, equity, inclusion, and justice within academia, specifically in the biological sciences.
Our goal is to provide an environment for continued learning, critical discussion, and brainstorming action items that individuals and labs can implement. Our discussions and proposed interventions reflect our own opinions based on our personal identities and lived experiences, and may differ from the identities and experiences of others. We will recap our discussions and proposed action items through a series of blog posts, and encourage readers to directly engage with DEIJ practitioners and their scholarship to improve your environment.
November 4th, 2022 – Blinding peer review
Discussion Leader: Eric Greene
Articles:
Summary Article: “Funding: Blinding peer review”
Primary Article: “An experimental test of the effects of redacting grant applicant identifiers on peer review outcomes”
Bonus Article: “Strategies for inclusive grantmaking”
Summary and Key Points:
STEM research funding is a highly competitive space that has a persistent lack of diversity and representation, especially at the faculty level. I chose this case study as it discusses one of the largest current racial disparities in STEM, highlights a source of white privilege that directly impacts lab funding, and provides experimental evidence towards one mitigation strategy.
The NIH is a substantial funding source for biomedical research in the US and NIH funding is foundational to the existence of many laboratories that are driving biomedical scientific discovery. However, there is a large and persistent funding gap between White and Black investigators, where Black PIs are funded at 55-60% White PIs rate.
In response to this disparity, the NIH conducted a study on the effects of blinding applicants’ identity and institution on the review of R01 proposals. The goal of this large experiment was to gain an understanding about the role of peer review in facilitating racial bias in grant awards and to understand the extent to which blinding applicant identity could blunt racial bias. The experiment uncovered the following:
Scores for applications from Black PIs were unaffected by blinding, but scores for applications from White PIs were significantly lower when the White PIs identity was blinded such that the racial gap was cut in half. This finding could be due to the “Halo effect” where personal/institutional prestige dramatically upweights advantaged/privileged individuals and can be seen as another mechanism fueling a ‘winners keep winning’ phenomena. Indeed the “Halo effect” has been indicated to be a potent factor in manuscript peer-review.
The principle critique of invoking the “Halo effect” to rationalize the findings of this study is that proposal writers did not write their proposal with identifying information redacted, it was done administratively with previously reviewed R01 applications, leaving uncertainty regarding the impact of administrative redaction on ‘grantsmanship’. However, we discussed the likelihood that applicants who benefit from individual/institutional prestige would likely write favorably toward this status in their applications thus in effect working to entrench any positive “Halo effect” benefit.
Blinding applicant identification on grant proposals is not a silver bullet that solves racial disparity in NIH funding. Including being imperfect itself, with ~22% of reviewers able to positively identify blinded applicant identity. However, this is one tool that has a demonstrated effect here to blunt reviewer bias. While blinding was somewhat effective here, there are means of double blinding and/or tiered blinding of application materials that can be used instead that may hold greater potential.
A key part of our discussion was about the review criteria for NIH funding that explicitly required a numerical evaluation of the individual and institution. Evaluation of a person contributes to an obligate entanglement of one’s past scientific accomplishments with their future potential during the grant review process. Not only can this equivalence be false (people often can succeed past initial setbacks), but it also can be harmful by promoting an applicant’s self-worth to be tied to their productivity. Funding requires accounting for equipment available to carry out the research, which is important for accountability on the part of the investigator, but does not necessarily require a numerical number. This detailed level of evaluation would prompt reviewers to score prestigious/well-resourced institutions higher even if the same research could be carried out elsewhere. We discussed as an alternative whether equipment/facilities categories could be scored as ‘sufficient’ or ‘insufficient’ and not influence the overall impact score of the application.
Open Questions:
Proposed Action Items:
While trainees may have limited influence to change the course of NIH peer review, there are nonetheless actions that one can take: