Like many others, I was disappointed with the lack of code, or even executables accompanying the publication of AlphaFold3 in Nature. This made it impossible to test the most exciting claim of the paper: impressive performance predicting the structures of proteins bound to novel ligands. I was even more upset to learn that my colleague Roland Dunbrack was “ghosted” after he submitted his initial review.
We organized an open letter to Nature, questioning why a journal would fail to enforce its written policies. In doing so, Nature implies that it enforcess those policies inequitably and to the detriment of the overall scientific community.
In response to this letter, Deepmind announced they would release the code in 6 months. This was a reversal from their previous quote (from a Nature News article):
””““We have to strike a balance between making sure that this is accessible and has the impact in the scientific community as well as not compromising Isomorphic’s ability to pursue commercial drug discovery,” says Pushmeet Kohli, DeepMind’s head of AI science and a study co-author.”””
Now Nature has replied in an unsigned editorial, saying it wants to be “in conversation” with our community around ensuring openness of the research ecosystem. This editorial focuses a lot on code disclosure. Journals want to play an important role in the research ecosystem going forward and have established that they will perform some valuable services:
The editorial makes it seem like this matter is only about a small execption regarding the code. And writes that the release of the code after a 6 month delay, spurred by the open letter and community outcry - not Nature, is:
“… an important step, and Nature will update the published paper once the code is released.”
Yet still, we cannot validate the most fundamental claims about protein ligand predictions. I find Nature’s description of the server disingenuous:
“The basics of how the community can use the new version of AlphaFold remain the same: anyone with a Google account can use the tool for free, for non-commercial applications.”
and
“In addition to the non-availability of the full code, there are other restrictions on the use of the tool — for example, in drug development. There are also daily limits on the numbers of predictions that individual researchers can perform.”
The server is restricted to 20 natural metabolites and ions. We still cannot even reproduce the figures of the paper.
Obviously, many companies want the Nature “stamp” of approval - this editorial shows, nakedly, that this “stamp” is a toxic part of our current research ecosystem, one that bends easily to corporate interests and applies inequitable standards. The canard that the private sector won’t publish if they don’t let companies play by different rules is particularly problematic. Nature broke their peer review process here (see Roland Dunbrack’s experiences) and with a little bit of community pressure, the authors changed course and promised an eventual code/executable release.
What’s the solution going forward? We can raise the bar! Academics should push the envelope in data and code disclosure alongside preprints with open review. Companies can also lead by example (see Arcadia Science, Pat Walters, and others) by doing a 1st class job of disclosing data outside journals.
I’m optimistic about the scientific ideas presented in the AF3 paper. It’s an exciting time for AI and biosciences. Let’s make the future get here faster by building on each others work!