Why Hasn’t Preprint Peer Review Caught On?
Preprints have transformed how we share science. Posting a preprint is joyful: you upload your work, and it just works. That frictionless experience is a big reason preprinting has become such a success.
But preprint peer review hasn’t taken off in the same way. This week, I realized one surprisingly simple reason why: authors often don’t even know when feedback happens.
The Notifications Are Missing!
When we give feedback on preprints (all of our reviews are posted here), I used to use Disqus, which is embedded into bioRxiv. More recently, we’ve been posting through PREreview (including in the graduate-level class we teach on peer review: details here).
But I’ve discovered that the authors never find out because:
- On bioRxiv, notifications for Disqus comments are opt-in. Many authors never enable them.
- On PREreview, there are no notifications at all unless an author explicitly pre-requests a review.
This means thoughtful, constructive reviews can easily go unseen. I only realized how widespread this issue is after running into an author who had never seen the review my students had written of their preprint. They loved the feedback once they did, but only because I mentioned it to them in person.
bioRxiv leadership has explained their caution to me: some authors don’t want unsolicited comments, and auto-notifications could feel like spam. But the result is that reviews are written… and no one hears about them.
I think that posting a preprint is already an explicit invitation for feedback. Authors should not have to “opt in” to learn that constructive, ORCID-authenticated reviews exist — especially when those reviews are linked right in the preprint’s sidebar on the BioRxiv website.
For now, our workaround will be to email authors directly when we post reviews. But this friction slows down the cultural change toward open, constructive feedback.
If we want preprints to truly catalyze faster, more open science, then our infrastructure needs to reduce barriers — not create them. Otherwise, we risk shouting into the void while authors miss out on the joy of receiving thoughtful feedback.
Three new papers from Roche, accompanied by a perspective by Elspeth Garman, describe a fantastic ligand–protein structural dataset. They focus on several related fatty acid-binding proteins (FABPs) and report 229 high-resolution ligand-bound structures. The first paper explores how protein dynamics (and allosteric interactions with membrane mimics) change when empty lipid-binding sites are filled by natural or synthetic ligands. The second examines detailed atomistic interactions across isoforms, with lessons for selectivity. Along the way, there is great crystallographic lore: nearly isomorphous crystals, twinning, and more. The third paper focuses on how ligand chemistry can often be mis-assigned due to complexities in synthesis, isomerization, or transformation within the crystal. This is a problem we have observed ourselves in our macrodomain ligands and I expect to see more often as “make on demand” chemistry democratizes ligand soaking.
The authors distill these findings into a set of very conservative lessons: prioritize chemical certainty, gate on full occupancy, and filter out ambiguous cases before using structural data for machine learning. Elspeth echos some of these concerns in the perspective. I disagree. I think we need to consider these lessons differently depending on the use case:
- Downstream medicinal chemistry or mechanistic interpretation without a structural biologist in the loop: If someone is simply taking the PDB as “truth” to guide synthesis, then rigorous filtering to avoid incorrect ligand identities is absolutely appropriate. I don’t think this actually happens anywhere, but this is always the straw man that such papers “warning” about the pollution of the PDB are concerned about. This includes papers criticizing our work (see also our response).
- Medicinal chemistry fully integrated with structural biology: Here, strict filtering risks losing huge opportunities. Alternate conformations in both ligand and binding site, unusual B-factors, and subtle occupancy differences are not nuisances. These are the very signals that can inspire new design strategies and suggest unexplored mechanisms of selectivity.
- Input to machine learning to predict protein ligand complexes: Curating only “gold-standard” data is going to be incredibly limiting. Disagreements between models and experimental data are opportunities to improve algorithms and better understand real-world uncertainty. I doubt that many of the structures that will be deposited by the OpenBind consortium will pass these filters.
Real experimental data is messy, full of alternate conformations, unexpected chemistries, and crystallization “oddities”. Filtering exclusively for perfection may feel safe, but it also limits discovery. Even though using coordinates of the partial occupancy ligands and static alternative conformations will improve things, I’m hoping that the ML for structural biology field will increasingly embrace the mess of experimental data more directly. I’ve written about this before from conceptual, practical, and policy perspectives. While this trio of papers represents a tremendous teaching text that guides the reader through many of the complexities of protein-ligand data sets, I disagree with the jeremiads at the end of these papers about the potential for misuse. I truly wish there were more careful papers like this out there.
We chatted about the latest news, including the new NIH open access policy, trends observed with scientists using LLMs, & the Montpelier Mile race.
“The new Administration began weaponizing what should not be weaponized - the health of all Americans…creating chaos and promoting an…unreasoned agenda of blacklisting certain topics, that…has absolutely nothing to do with the promotion of science.” - Judge William Young (reported by Max Kozlov, Nature News)
The fourth episode of The Tortured Proteins Department is out now!
We chatted about the latest news, our Conformational Ensembles Conference, vibe coding in science, and the Dipsea and Montpelier Mile races.
The pre-prints discussed in this episode: