Eiko Fried is a psychologist, methodologist, open science enthusiast, an Associate Professor in the Department of Clinical Psychology at Leiden University, and Associate Editor for UC Press’s open-access journal Collabra: Psychology, which is the official journal of the Society for the Improvement of Psychological Science (SIPS). In this Peer Review Week 2022-themed interview, we catch up with Dr. Fried on peer review and Collabra’s Streamlined Review submission process.
UC Press: Happy Peer Review Week! The theme of this year’s Peer Review Week is “Research Integrity: Creating and supporting trust in research.” How does open science contribute to trust in research?
EF: Most of us researchers collect empirical evidence, and then we draw inferences from the evidence. Drawing appropriate inference is not only important because we want to get things right, but because our research findings can have important implications: a medication may be approved, or a government may implement a costly public health initiative.
One of the core pillars of open science is transparency. I believe that transparency is crucial to help us evaluate whether inferences researchers draw in their papers follow from the evidence presented, and I will present three scenarios to make my case. First, researchers may act suboptimally in ways that threaten valid inferences, and I’m ok to assume that for most of these cases, this happens without nefarious intent. For example, it is quite common that researchers analyze data in various ways, but then only report the analyses in which significant results were obtained. This is problematic, and having transparency about all analyses helps mitigate problematic inferences: if I know a researcher engaged in questionable research practices such as selective reporting, I will be more skeptical about their inferences.
Second, transparency helps in catching honest mistakes. Researchers are humans, and no matter how careful, mistakes can happen. For example, I made a mistake in my code for a paper published in 2016. But because I have shared all my code for all papers since I learned about open science initiatives in 2015, another researcher was able to identify that mistake, alert me about it, and I could publish a correction (I wrote about this experience here in some detail).
Third, there is also nefarious intent, and some researchers knowingly engage in scientific malpractice. They tamper with or falsify data. Luckily, this kind of behavior is rare, but it does thrive in research environments without transparency.
You asked how open science contributes to trust in research. Openness is necessary to allow the research community, journalists, policy makers, and other stakeholders to evaluate research. Without transparency, only the researchers who carried out the research can evaluate their own findings, and we all agree that this is not desirable.
UC Press: One somewhat novel aspect of Collabra’s peer review process is open peer review. What is open peer review?
EF: Open peer review means that if a paper is published in Collabra, the peer reviews are published along with the manuscript. Note that open peer review is not the same as signed peer review: at Collabra, reviewers can decide to sign their review, but they do not have to. If a paper is not published, the reviews are not open by default, but the journal allows researchers to share these reviews, e.g. in case they want to submit the paper to another journal and show that they appropriately responded to prior reviews. This is important because many journals treat reviews as confidential (that is, they tell me I cannot share the review I myself wrote), although the legal status of this is not entirely clear to me.
Open peer review ties into the topic of transparency we just discussed, and helps keep the journal (represented by the editor and peer-reviewers) accountable. I have reviewed several manuscripts in high-impact psychiatry journals in the last few years which were just “waved through” by all other reviewers (“This is an excellent paper, I have no comments”), clearly without having critically engaged with it. As a reader (policy maker, journalist, etc), I would like to know if a paper was critically reviewed or not; open review helps with that. Similarly, an editor at Collabra may make a mistake, or base their rejection decision on a flawed review. Collabra holds itself accountable for such mistakes with open peer review.
UC Press: In contrast to sketchy research getting “waved through,” some journals will reject papers, not based on the quality of the research or the soundness of their methodology, but because they think the research isn’t novel enough or they predict the research won’t be impactful. Collabra has developed a “fast track” submission process (Streamlined Review) for previously-rejected papers. How does this work, and what has been your experience with these papers?
EF: A few years back, an editor rejected my paper, after two rather positive reviews and one very negative review. I decided to update my manuscript, take the feedback into account, and submit the paper to another journal. I also attached the prior reviews and how I responded to them: this way, I could show that I had taken the comments of the first two reviewers seriously, and that the third reviewer had misunderstood the paper. It was a bit of a gamble, and I had never heard of doing this before, but it worked out: the editor invited only one additional reviewer, and the paper was published eventually.
A few years later, I learned about the fast track in Collabra, and it works pretty much exactly like that. If your paper is rejected elsewhere based on reasons other than thorough science (e.g. because Nature wants research results to be “novel” and “surprising” rather than “rigorous” or “robust”), you can submit your paper to Collabra, along with all prior relevant correspondence, e.g. decision letters from the editor, or reviewer comments if you can share them. These papers may still be desk-rejected, of course, if the handling Collabra editor does not judge them to be scientifically sound. But in the best case for the authors, the paper can be accepted without further peer-review, given that the paper may have already been peer-reviewed, all important points regarding scientific may have been addressed already by the authors, and the only thing that stood in the way of publication in another journal are non-scientific soft factors such as how sexy a result is.
UC Press: Thank you so much for sharing your experiences with peer review, and for your work with Collabra!
EF: Happy peer-review week 😉 !
Related: Peer Review Week 2021: A Q&A with Collabra: Psychology Editor-in-Chief Simine Vazire
Collabra: Psychology—the official journal of the Society for the Improvement of Psychological Science—is a mission-driven, open-access journal from University of California Press that shares not only the research it publishes, but also the value created by the psychology community during the peer-review process. Collabra: Psychology has seven sections representing the broad field of psychology, including a highlighted focus area of “Methodology and Research Practice.”
online.ucpress.edu/collabra
Twitter: @CollabraOA