See also...
Claim: Rating and feedback is better than an âall-or-nothingâ accept/reject process. Although people like to say âpeer review is not binaryâ, the consequences are.
âPublication in a top journalâ is used as a signal and a measuring tool for two major purposes. First, policymakers, journalists, and other researchers look at where a paper is published to assess whether the research is credible and reputable. Second, universities and other institutions use these publication outcomes to guide hiring, tenure, promotion, grants, and other ârewards for researchers.â
Did you know?: More often than not, of the "supply of spaces in journalsâ and the âdemand to publish in these journalsâ. Who is the consumer? Certainly not the perhaps-mythical creature known as the âreaderâ.
Perhaps not. We are accustomed to using ratings as filters in our daily lives. Readers, grantmakers, and policymakers can set their own threshold. They could disregard papers and projects that fail to meet, for instance, a standard of at least two peer reviews, an average accuracy rating above 3, and an average impact rating exceeding 4.
In the field of economics, between the âfirst working paperâ that is publicly circulated and the final publication. During that time, the paper may be substantially improved, but it may not be known to nor accepted by practitioners. Meanwhile, it provides little or no career value to the authors.
As a result, we see three major downsides:
Time spent gaming the system:
Researchers and academics spend a tremendous amount of time 'gaming' this process, at the expense of actually doing better research.
Randomness in outcomes, unnecessary uncertainty and stress
Wasted feedback, including reviewer's time
I (Reinstein) have been in academia for about 20 years. Around the departmental coffee pot and during research conference luncheons, you might expect us to talk about theories, methods, and results. But roughly half of what we talk about is âwho got into which journal and how unfair it isâ; âwhich journal should we be submitting our papers to?â; how long are their âturnaround times?â; âhow highly rated are these journals?â; and so on. We even exchange on how to
There is a lot of pressure, and even bullying, to achieve these âpublication outcomesâ at the expense of careful methodology.
The current system can sideline deserving work due to unpredictable outcomes. There's no guarantee that the cream will rise to the top, making research careers much more stressfulâeven driving out more risk-averse researchersâand sometimes encouraging approaches that are detrimental to good science.
A lot of âfeedbackâ is wasted, including the . Some reviewers write ten-page reports critiquing the paper in great detail, even when they reject the paper. These reports are sometimes very informative and useful for the author and would also be very helpful for the wider public and research community to understand the nature of the debate and issues.
However, researchers often have a very narrow focus on getting the paper published as quickly and in as high-prestige a journal as possible. Unless the review is part of a 'Revise and Resubmit' that the author wants to fulfill, they may not actually put the comments into practice or address them in any way.
Of course, the reviews may be misinformed, mistaken, or may misunderstand aspects of the research. However, if the paper is rejected (even if the reviewer was positive about the paper), the author has no opportunity or incentive to respond to the reviewer. Thus the misinformed reviewer may remain in the dark.
The other side of the coin: a lot of effort is spent trying to curry favor with reviewers who are often seen as overly fussy and not always in the direction of good science.
John List (Twitter : "We are resubmitting a revision of our study to a journal and the letter to the editor and reporters is 101 pages, single-spaced. Does it have to be this way?"
Paola Masuzzo; âI was told that publishing in Nature/Cell/Science was more important than everything else.â
Anonymous; "This game takes away the creativity, the risk, the âright to failâ. This last item is for me, personally, very important and often underestimated. Science is mostly messy. Whoever tells us otherwise, is not talking about Science.â
Here's a rough sketch of the process and timings at top journals in economics. report an average of over 24 months between initial submisson and final acceptance (and nearly three years until publication).