A few years ago I started signing my manuscript reviews. My major motivation was to give the authors an opportunity to contact me to clarify what I meant or to ask how strongly I felt about statements without the delays of multiple rounds of review mediated by an editor. I figured that the positives for increasing the clarity, transparency, and speed of scientific communication would outweigh any potential negative side effects, such as “retribution”. Another positive was that I had to write each review thoroughly and fairly.
Despite some weird dynamics with editors, authors, and other reviewers throughout the years, I’ve found this to be a positive experience overall. But still had two nagging issues with peer review. First, I’ve had experiences where my reviews have been ignored after rejection at Journal #1 and papers have ultimately appeared in Journal #2 (or #3….) without ever responding to the substantive issues that I or other reviewers raised. Second, I was still bothered that we lose a tremendous amount of the written scientific record because reviews are only ever seen by the editors, authors, and reviewers themselves. This second issue is somewhat mitigated at journals like EMBO, Nature Communications, and eLife, which have more transparent review processes for accepted papers. And as part of ASAPbio, we are working on spreading this idea further in our initiatives on peer review
But, in the background of these larger initiatives and reflecting back on my own reasons for preprinting our lab’s work, I think there is more that we can do as individual scientists. Earlier this year, I decided to only review papers that have preprints associated with them so that I could post the reviews as comments on the preprints. I’ve openly reviewed 6 papers in the last year:
By far the best interaction has been with Sagar Khare’s lab, where they responded to our review on BioRxiv (and through Nature Chemistry, which eventually invited us to write a News and Views on their accepted article). It’s also been instructive to see that sometimes the authors interpret a remark very differently than the way I intended it and we can clarify the point directly. I also like the idea that the authors can start thinking about our comments immediately, instead of waiting for all the reviewers and the editor to collate things together. Another side benefit is that I can work more efficiently: the proliferation of Elsevier Editorial System accounts overwhelms my ability to use a password manager and now I can just email the editors a link to my comment.
The weirdest interactions I’ve had has been with some journals who worry that I’m breaking the confidentiality of the review process (for example). The authors have already posted a preprint - so I’m not sure about my responsibility to keep their manuscript confidential as it is already public. Similarly, I’m confused as to whether posting a public comment on a public manuscript is the same as corresponding directly with the author (presumably the editors are worried about backroom deals, not public comments)?
Since adopting this stance, my reviewing load has been substantially lower. I’m considering just picking some preprints to review with lab members (even when not asked by a journal to review them) and posting our reviews. Hopefully the authors can incorporate the feedback and add it to their portfolio when they are seeking journal sanctioned peer review.
The etiquette of commenting on preprints is still developing. By posting manuscripts on a preprint server, are authors inviting the same type of scholarly feedback that now occurs behind closed doors? I’m operating under the assumption that authors would welcome this type of feedback. As a reviewer, posting our reviews gives us the same benefits that drew us to posting preprints in the first place: transparency, speed, and scientist-control of disclosure!
Finally, I recognize that I can do this because of my secure position at UCSF. We need mechanisms to help folks who aren’t in as secure a position to be able to contribute in an anonymous, but somehow credited way, as well. My hope is that by moving the scientific communication culture towards something that is more open and transparent, potential for retribution will be reduced and the need for anonymity will be decreased.
Last Friday saw Rahel’s thesis talk and a goodbye party at Spark for Brandi. Rahel moves down to Stanford as a postdoc, where she will be exploring the exciting developments in cryo electron tomography in Wah Chiu’s lab. Brandi will move to Boston to take a Scientist position at Relay Therapeutics, a company focused on exploiting protein dynamics for drug discovery. Brandi was already pretty familiar with Relay as she had been collaborating with them since joining the lab in 2016!
It’s been over a year since I wrote an update for our lab news section (parenthood has slowed me down a bit). There have been a lot of changes:
Goodbye David Mavor, PhD!
David (aka Iggy) finished his doctoral work in the lab and moved on to Dan Bolon’s lab at UMass Medical. Iggy merged his thesis work with graduate education in the PUBS course and published a spectacular deep mutational scanning paper (with a record number of co-authors!) based on the the first year of the class. The next two years were summarized in a pre-print. Congratulations Dr. Mavor
Welcome to new postdocs Dr. Brandi Hudson (and soon Dr. John Lee)!
Brandi joined us last summer from the Tantillo lab at UC Davis and has been working in collaboration with Relay Therapeutics and Schrodinger on how to account for conformational dynamics in ligand design. John will be joining us later this month from the Burkart lab at UCSD.
New rotation students!
Two fantastic students are rotating with us this quarter. Jenna from iPQB and Robbie from Tetrad. Jenna is working with Erin Thompson on compensatory mutations in enzymes and Robbie is working with Ben Barad on chitinases!
Foreign visitors to the lab!
During 2016 we had Tommy Lazarou visiting from Canada as part of his co-op program at McMaster University. He’s now a graduate student at NYU. Dr. Kazutaka Ito recently joined us from Asahi KASEI Pharma Corporation in Japan and will be working with Michael Thompson on proteases and XFELs (including our trip later this month to the SACLA XFEL in Japan). Early next year, Prof. Roberto Chica will be joining us on sabbatical from University of Ottawa to work on room temperature X-ray diffraction studies of designed enzymes. I love the international community of science. Visiting the Tawfik lab at the Weizmann Institute in 2010 was a formative experience for me, so it’s fun to host people here and continue the tradition of scientific exchange!
Goodbye Professor Daniel Keedy!
Postdoc Daniel Keedy accepted a tenure track position in New York, NY at the ASRC of CUNY as part of their Structural Biology Initiative. We held his goodbye party this week and he has already started his long drive across the country. In our lab, supported by the A.P. Giannini Foundation, Daniel pioneered new computational and experimental approaches to “map the intramolecular nervous systems of proteins” - a synthesis we’ve come to call multiconformer multitemperature X-ray crystallgoraphy (or MMX). Daniel always had a remarkable way with words and his attention to detail was second to none. He took a big chance coming to join a fresh new lab and I’m so proud of all that he accomplished here. His last manuscript is almost ready to post on BioRxiv and we’re excited to see how Prof. Keedy adjusts to being the “big cheese” of the Keedy lab in 2018.
Starting a new “Macromecular Structure Methods” class!
John Gross and I have started a new class for UCSF Biophysics and Chemical Biology students. In the tradition of PUBS, this new Methods class has a project-based format. Organizing the material has invovled a lot of heavy lifting by David Bulkley (in our EM core), Ryan Tibble (in John’s lab), Eugene Palovcak and Daniel Asarnow (in Yifan Cheng’s lab), and Kazu Ito and Michael Thompson (in our lab!). It is a lot of work to start these kinds of classes, but I think it is worth it. I’ll update with an assessment later in the year!
At the ASAPbio meeting, I talked about the importance of building a lab culture as a junior faculty member. Two of the most important things about the culture we are trying to build in the Fraser lab are: 1) we have high scientific standards and 2) we enjoy communicating our results. Preprints help us with both. We set a high standard by showing the world our results when we are ready by posting a preprint, not after weeks or months in review. We also do this because we hope that other scientists will give us feedback and build on our work. We celebrate posting the preprint (which we do simultaneously upon submitting to a journal, but more on that later) because we want to congratulate each other on the hard work that went into the paper. Preprints return the control over both of these important aspects of culture to us, the scientists.
In articulating these aspects of our lab culture, I also mentioned our lab motto: Beer and Tacos. Now that this phrase has made the New York Times, I thought I might explain the history of “Beer and Tacos”. My attraction to this phrase has its origin, like many of my scientific interests, in baseball statistics. In 2003 (when I was an undergraduate at McGill), Dayn Perry used the “Beer and Tacos” analogy in describing the false dichotomy between data driven analytics (sabermetrics/statistical analysis) and observational scouting (which is more qualitative). Perry’s story was in part motivated by a grumpy article written by my hometown baseball writer Richard Griffin. Although I’ve never actually confronted Griffin about it, I think I may have played a small part in Griffin’s motivation for writing the article. A few days before his article appeared, I was on the field at the SkyDome prior to a Blue Jays game filming a brief segment for the Discovery Channel Canada on baseball statistics (unfortunately, I can’t find the video online!). Griffin walked by, shook his head, and continued on his way. Griffin hated analytics and loved scouting.
At the time this was a bit of a holy war. It is dramatized in the movie Moneyball, based on the excellent book by Michael Lewis. In recent years, the value of analytics has been proven time and time again not only in baseball, but also in other sports and even in finance, marketing, etc. However, part of the reason that analytics keeps permeating more and more areas is that there is an intense drive to capture the qualitative aspects and turn them into something quantitative. It is no longer sufficient for a scout to say that an outfielder “tracks the ball well”. This skill wasn’t captured that well in 2003 by simple counting statistics like range factor, but now we can plot the actual path the player takes to determine whether he takes the most efficient route to the ball. What does “beer and tacos” mean? Dayn Perry said it best. In responding to Richard Griffin and others who argued that organizations such as the Boston Red Sox, run at the time by Theo Epstein, should focus less on stats and more on scouting, Perry wrote:
“Grumps like Griffin don’t understand the concept of synergy. A question that’s sometimes posed goes something like this: “Should you run an organization with scouts or statistics?” My answer is the same it would be if someone asked me: “Beer or tacos?” Both, you fool. Why construct an either-or scenario where none need exist? Heady organizations know they need as much good information as possible before they make critical decisions. Boston under Epstein, for example, is a veritable clearinghouse for disparate ideas and perspectives, and so far it’s working just fine.”
From 2005-2010, I was in graduate school at UC Berkeley. When my graduate advisor, Tom Alber, would suggest multiple experiments during group meeting, I would often yell out “Beer and Tacos” from the back of the room. The idea being that we shouldn’t prioritize only experiment A or experiment B, but that we should probably consider doing both. In 2011, when I started my lab, I continued shouting “Beer and Tacos” in my own group meetings at UCSF. We want to create “a veritable clearinghouse for disparate ideas and perspectives”.
Now we come to publishing. People like Mike Eisen argue for BEER (only immediate publication/preprints and post publication peer review/PPPR) and people like Emilie Marcus argue for TACOS (only traditional journals, unless you have a special “wink wink” agreement). The system is evolving. It’s clear that “journals only” is not going to be the future. The delays and politics around “formal peer review” can be detrimental to our goals as scholars. However, many of the trends on how journals are coordinating peer review are all about openness, speed, and scholarship. It’s also clear that journals can add value, even beyond the obvious benefits in coordinating peer review: reading the table of contents is (surprisingly) the nearly exclusive way that many scientists become aware of work outside their lab. Becoming aware of papers in this manner is not the ideal system in the long term, but it does add value today. The exact pairing of preprints and journals is in flux - but there is wide agreement that we value scholarship and rapid communication of results.
As we debate exactly what type of beer we will be drinking in the future and what filling should go in the taco, my lab cares about getting our work out there as quickly as possible and ensuring that it reaches the widest audience possible. So we ignore shouts for “Beer OR Tacos” and choose “Beer AND Tacos”. When we post preprints, we are happy about getting our work out there immediately, we are eager to improve the work through peer review in any form, and we are hopeful that eventually publishing in a traditional journal will help it find an audience. And so we go to celebrate, as a lab, by ordering “Beer and Tacos”.
