The last few years has seen an explosion in the use of preprint servers to circulate research results, especially in the life sciences. Long established in physical and mathematical sciences, it became more prevalent in recent years in biology. This is explained in part by a burgeoning critique of the “gatekeeping” role of mainstream high impact-factor journals, and accelerated particularly during the COVID-19 pandemic when conventional journals could not keep up with the pace of new results.
In turn, this has also been accompanied by a more broad-based critique of the peer review process itself. Traditional peer review is seen as slow, opaque, and biased. Some see it as condescending – why do we need someone else to tell us what counts as good science? I recently stumbled across an interesting blog which points out, quite reasonably, that peer review is recent in the history of science. It was only in the early 20th century that it became somewhat widespread, and only after around the 1960s did it become the “default” route to publication for most researchers. I welcome this sort of perspective, as it is easy to forget that science is not just an abstract idea, but also an evolving set of social institutions with a relatively short modern history. To be reflexively conservative about society’s ancient institutions is one thing, but to stand on ceremony for arrangments not old enough to claim a pension is quite another.
This is all well, but recasting peer review as a recent, progressive aberration leaves some questions. Why did it emerge, and why was it taken up so enthusiastically in the first place? What are we going to do about it if we decide that peer review is not the best way to evaluate scientific work? What are the alternatives?
I think we have to think hard about what the options on the table are. There are, I believe, two basic perspectives. One is based on maximal optimism about collective rationality (and dilligence). Akin to the Hayekian argument in favour of free markets over central planning, it assumes that people “on the street” with immediate skin in the game are in the best position to judge what is worth investing time in or not. Scientific communities can be trusted to monitor the literature and organically elevate the most significant findings. In a limited way, I’d say something like this has to be at least partially true for science as an enterprise to work at all (which, mind you, we might be skeptical about). But acknowleding there may be more to our decision making than what is maximally rational might lead to some other ideas. In this regard, the other option is more clear eyed about our intellectual frailties. This holds that, outside the most niche areas, in the 21st century no scientist has the attention span to keep abreast of the entire literature. Scientists always rely on shortcuts to sift through and decide what merits their attention. If these shortcuts involve heuristics about reputation, loyalty, or professional self-interest, we will surely be at risk of ignoring meritorious work, or missing flaws in problematic work. We will also be at risk of reproducing power dynamics within and between institutions, and even nations (in my own experience, the frequency with which casual racism is used as a “heuristic” to dismiss perfectly good work speaks to this).
I do not really accept Hayekian arguments about the economy, but they at least have the virtue of being able to appeal to straightforwardly aligned incentives: agents in an economy are incentivised to improve their own welfare, and in a market economy, one can draw a plausible line between having accurate information and best improving one’s welfare. In science, this sort of line can typically only be drawn when “accurate information” corresponds to “what the scientific community will be positively disposed towards”, because the welfare of a scientist depends primarily on how they are perceived within their community. It is hoped - and again, to a degree necessary under any pro-science view - that the community is generally more positively disposed towards what is true. But anyone can see that is quite different to argue that incentives keep us right in the market since we will get rich if we have novel, useful information (or go hungry if we’re wrong), versus arguing that incentives keep scientists right because they’ll get famous if they discover something (or the community will find out and demote them if they’re wrong). We’d have to assume scientific merit is the only factor deciding the community’s reaction to new findings, which stretches credulity. There is clearly scope for suboptimal equilibria where incentives diverge. What if scientists are more inclined to believe work that is consistent with that of their friends or perceived superiors? What if professional advancement depends on networking and being liked? Do you want to offend the top person in your field, who might decide whether you advance in the field, by siding with some nobody whose work goes against theirs?1 Any scientists reading this will hardly need resort to imagination.
If we take seriously that cognitive and social biases can have a signficant impact on the spread of knowledge in science, we might ask whether there is something to be done. One idea is usually another favourite among the more “free market” inclined – division of labour. Instead of leaving it to people mired in technical minutia to self-assign time for skimming through hundreds of unedited manuscripts in spare, tired moments, offer a service whereby it becomes someone’s job to give their full attention to the task of carefully reading manuscripts, without regard to their nation of origin or professional standing. Have editors who know how to write well help polish up rough texts, and implement a consistent visual language so that people aren’t penalised for being poor designers (or benefitting because they’re talented artists). Make a system of blinded expert peer review where adopting a critical perspective is encouraged through anonymity. We can start to see how the argument for peer review might be made. Presumably, the critics are well versed in these arguments (although they are often apt not to address them). It should be clear that a form of systematised peer review is promising to solve at least some problems, even if, in the final analysis, we come out against it.
Peer review in abstract might be a great idea, but peer review as it is actually implemented is open to all sorts of additional criticism. In my view, the main flaws of peer review are in how it fails to address the biases I mentioned before. Peer review is not double-blinded in most journals, and this has demonstrable effects on publication success. Peer reviewers are not in any way rewarded for their time investment, so they often do not put enough effort into the process for it to be worthwhile. Nothing about the peer review mechanism prevents a scientist assigned to peer review from intentionally torpedoing a result that contradicts their own (or potentially scoops them), or that they suspect might be from a competitor lab (even with double blinding, it is usually possible to guess who submitted an article, especially in small fields). But if critics agree that these are flaws of peer review, how are they not also even greater flaws of no-peer-review?
None of the above really gets to the heart of the matter though. There is one fundamental factor that I suspect means peer review is here to stay: demand. While peer review is often counterposed with laissez-faire, it is actually a product of laissez-faire.
Stop complaining about peer review and just stop publishing in journals entirely.
— Ryan Rhodes ⚙️🧠 (@cogscidude) March 1, 2024
Just write what you want to write and put it online–in a blog, on a preprint server, who cares. The information is what matters, and there are better ways to disseminate it than a journal!
This Tweet from Ryan Rhodes is interesting because it seems basically right. If you’re a critic of peer review, why don’t you just boycott it? There is nothing mandatory about it – it’s just that it happens to be the preferred way for most researchers to get their research nowadays. Why? Because the literature is an endless torrent and there is an appetite for someone to come along and say: these are the things that are worth reading. To harken back to the old days of science where people didn’t bother about peer review is to ignore that the literature is several orders of magnitude larger and more complex now than it was then. There was a time when an immunologist could read every immunology paper published on a given week, giving each their full attention. This is no longer even close to being true, even in narrow specialisations. We are inundated, and we want some help.
An interesting example of how this organic demand for curation plays out has been documented in the AI/ML field recently. Two Twitter users @arankomatsuzaki and @_akhaliq have been Tweeting daily paper recommendations to (at the time of writing) ~90,000 and ~300,000 followers respectively while the AI landscape has been exploding in the wake of ChatGPT. Most of the work in question is published on pre-print servers like arXiv, as this has long been an accepted norm in computer science. Work by a team at University of Californa, Santa Barbara found that having your paper Tweeted out by both of these accounts is associated with a 2-3x higher citation count in the following period. The results cannot really establish causality (the study attempts to control for independent measures of quality, but this is no substitute for a causal inference design), but there is no question that the accounts in question are influential. Why wouldn’t they be? They have great reputations and they’re giving up large chunks of their time to help people.
What I think this speaks to is that when information overload sets in, people are always going to look for help to prioritise. If we are not systematic about this, we will likely be at the mercy of opaque algorithms, famous Twitter accounts, or simply our unconscious biases to distill the literature for us, because we cannot read it all. I think traditional journal peer review is at best a rough first draft of the system we might hope to have. Nevertheless, I do not really think there is a viable alternative that involves entirely abolishing curation (with all its attendent problems). The debate isn’t whether to be selective; it’s just about what the selection mechanism will be (or whether we even care).
Footnotes
Of course, many similar arguments can go for the economy too, if with a more subtle force, which is among the reasons I am not a Hayekian or indeed a free marketer.↩︎