Why pay evaluators (reviewers)?

It's a norm in academia that people do reviewing work for free. So why is The Unjournal paying evaluators?

From a recent survey of economists:

We estimate that the average (median) respondent spends 12 (9) working days per year on refereeing. The top 10% of the distribution dedicates 25 working days or more, which is quite substantial considering refereeing is usually unpaid.

General reasons to pay reviewers

Economics, turnaround times

The peer-review process in economics is widely-argued to be too slow and lengthy. But there is evidence that payments may help improve this.

In Charness et al's full report, they note that few economics journals currently pay reviewers (and these payments tend to be small (e.g., JPE and AER paid $100 at the time). However, they also note, citing several papers:

The existing evidence summarized in Table 5 suggests that offering financial incentives could be an effective way of reducing turnaround time.

Equity and inclusivity

The report cited above notes that the work of reviewing is not distributed equally. To the extent that accepting to do a report is based on individual goodwill, the unpaid volunteer model could be seen to unfairly penalize more generous and sympathetic academics. Writing a certain number of referee reports per year is generally considered part of "academic service". Academics put this on their CVs, and it may lead to being on the board of a journal which is valued to an extent. However, this is much less attractive for researchers who are not tenured university professors. Paying for this work would do a better job of including them in the process.

Incentivizing useful, unbiased evaluations

'Payment for good evaluation work' may also lead to fair and more useful evaluations.

In the current system academics may take on this work in large part to try to impress journal editors and get favorable treatment from them when they submit their own work. They may also write reviews in particular ways to impress these editors.

For less high-prestige journals, to get reviewers, editors often need to lean on their personal networks, including those they have power relationships with.

Reviewers are also known to strategically try to get authors to cite and praise the reviewer's own work. They maybe especially critical to authors they see as rivals.

To the extent that reviewers are doing this as a service they are being paid for, these other motivations will be comparatively somewhat less important. The incentives will be more in line with doing evaluations that are seen as valuable by the managers of the process, in order to get chosen for further paid work. (And, if evaluations are public, the managers can consider the public feedback on these reports as well.)

Reasons for The Unjournal to pay evaluators

1. We are not ‘just another journal.’ We need to give incentives for people to put effort into a new system and help us break out of the old inferior equilibrium.

2. In some senses, we are asking for more than a typical journal. In particular, our evaluations will be made public and thus need to be better communicated.

3. We cannot rely on 'reviewers taking on work to get better treatment from editors in the future.' This does not apply to our model, as we don't have editors make any sort of ‘final accept/reject decision’

4. Our ‘paying evaluators’ brings in a wider set of evaluators, including non-academics. This is particularly relevant to our impact-focused goals.