LogoLogo
  • The Unjournal
  • An Introduction to The Unjournal
    • Content overview
    • How to get involved
      • Brief version of call
      • Impactful Research Prize (pilot)
      • Jobs and paid projects with The Unjournal
        • Advisory/team roles (research, management)
        • Administration, operations and management roles
        • Research & operations-linked roles & projects
        • Standalone project: Impactful Research Scoping (temp. pause)
      • Independent evaluations (trial)
        • Reviewers from previous journal submissions
    • Organizational roles and responsibilities
      • Unjournal Field Specialists: Incentives and norms (trial)
    • Our team
      • Reinstein's story in brief
    • Plan of action
    • Explanations & outreach
      • Press releases
      • Outreach texts
      • Related articles and work
    • Updates (earlier)
      • Impactful Research Prize Winners
      • Previous updates
  • Why Unjournal?
    • Reshaping academic evaluation: Beyond accept/reject
    • Promoting open and robust science
    • Global priorities: Theory of Change (Logic Model)
      • Balancing information accessibility and hazard concerns
    • Promoting 'Dynamic Documents' and 'Living Research Projects'
      • Benefits of Dynamic Documents
      • Benefits of Living Research Projects
    • The File Drawer Effect (Article)
    • Open, reliable, and useful evaluation
      • Multiple dimensions of feedback
  • Frequently Asked Questions (FAQ)
    • For research authors
    • Evaluation ('refereeing')
    • Suggesting and prioritizing research
  • Our policies: evaluation & workflow
    • Project submission, selection and prioritization
      • What research to target?
      • What specific areas do we cover?
      • Process: prioritizing research
        • Prioritization ratings: discussion
      • Suggesting research (forms, guidance)
      • "Direct evaluation" track
      • "Applied and Policy" Track
      • 'Conditional embargos' & exceptions
      • Formats, research stage, publication status
    • Evaluation
      • For prospective evaluators
      • Guidelines for evaluators
        • Why these guidelines/metrics?
        • Proposed curating robustness replication
        • Conventional guidelines for referee reports
      • Why pay evaluators (reviewers)?
      • Protecting anonymity
    • Mapping evaluation workflow
      • Evaluation workflow – Simplified
    • Communicating results
    • Recap: submissions
  • What is global-priorities-relevant research?
  • "Pivotal questions"
    • ‘Operationalizable’ questions
    • Why "operationalizable questions"?
  • Action and progress
    • Pilot steps
      • Pilot: Building a founding committee
      • Pilot: Identifying key research
      • Pilot: Setting up platforms
      • Setting up evaluation guidelines for pilot papers
      • 'Evaluators': Identifying and engaging
    • Plan of action (cross-link)
  • Grants and proposals
    • Survival and Flourishing Fund (successful)
    • ACX/LTFF grant proposal (as submitted, successful)
      • Notes: post-grant plan and revisions
      • (Linked proposals and comments - moved for now)
    • Unsuccessful applications
      • Clearer Thinking FTX regranting (unsuccessful)
      • FTX Future Fund (for further funding; unsuccessful)
      • Sloan
  • Parallel/partner initiatives and resources
    • eLife
    • Peer Communities In
    • Sciety
    • Asterisk
    • Related: EA/global priorities seminar series
    • EA and EA Forum initiatives
      • EA forum peer reviewing (related)
      • Links to EA Forum/"EA journal"
    • Other non-journal evaluation
    • Economics survey (Charness et al.)
  • Management details [mostly moved to Coda]
    • Governance of The Unjournal
    • Status, expenses, and payments
    • Evaluation manager process
      • Choosing evaluators (considerations)
        • Avoiding COI
        • Tips and text for contacting evaluators (private)
    • UJ Team: resources, onboarding
    • Policies/issues discussion
    • Research scoping discussion spaces
    • Communication and style
  • Tech, tools and resources
    • Tech scoping
    • Hosting & platforms
      • PubPub
      • Kotahi/Sciety (phased out)
        • Kotahi: submit/eval/mgmt (may be phasing out?)
        • Sciety (host & curate evals)
    • This GitBook; editing it, etc
    • Other tech and tools
      • Cryptpad (for evaluator or other anonymity)
      • hypothes.is for collab. annotation
Powered by GitBook
On this page

Was this helpful?

Export as PDF
  1. Parallel/partner initiatives and resources

Economics survey (Charness et al.)

PreviousOther non-journal evaluationNextManagement details [mostly moved to Coda]

Last updated 1 year ago

Was this helpful?

Improving peer review in economics: Charness et al. project and survey

We designed and disseminated a survey taken by over 1,400 economists in order to (i) understand their experiences with peer review and (ii) collect opinions about potential proposals to improve the system.

...

We reviewed the existing literature about peer review, drawing on sources from inside and outside of economics. ... We then built a (non-comprehensive) themed bibliography,

... we took the additional step of preparing a list of over 160 proposals.


Other peer-review models Our current peer-review system relies on the feedback of a limited number of ad-hoc referees, given after a full manuscript was produced. We consider several changes that could be made to this model, including:

  • Post-publication peer review: Submissions could be published immediately and then subjected to peer review, or they could be subject to continued evaluation at the conclusion of the standard peer-review process.

  • Peer review of registered reports: Empirical papers could be conditionally accepted before the results are known, based on their research question and design. A limited number of journals have started to offer publication tracks for registered reports.

  • Crowdsourced peer review and prediction markets: Rather than relying on a small number of referees, the wisdom of crowds could be leveraged to provide assessments of a manuscript's merits.

  • Non-economists and non-academics as referees: Besides enlarging the size of the pool of referees who assess a paper, the diversity of the pool could be increased by seeking the opinion of researchers from other disciplines or non-academics, such as policy makers.

  • Collaborative peer review platforms: Communication between authors, reviewers, and editors could be made more interactive, with the implementation of new channels for real-time discussion. Collaborative platforms could also be set up to solicit feedback before journal submission occurs.

EVALUATING RESEARCHEVALUATING RESEARCH
Logo
DiscussionEVALUATING RESEARCH
Logo
ReportEVALUATING RESEARCH
Logo