Trial reporting template

For each proposed/ongoing/past trial, we should report the following minimal details, with links (proposed template)

For each proposed/ongoing/past trial, let's try to report the following minimal details, with links (proposed template) If you don't have time and you have another clear presentation of most of this, please link or embed it.

"Concise reporting template"

Please keep your answers brief -- if you want to give more detail (which is not necessary) please link a later section or external page. _

Short version of this template as Google Doc HERE (link copy-opens a new version for you to work in)

General idea, main 'hypothesis' (if there is one)

Firstly, what is this promotion trying to do (e.g., 'encourage signups for giving pledges')?

But more importantly, what are you trying to learn here... What might you have a better understanding of after the trial than you did before the trial?

E.g.,

Specifically:

• Does the opportunity we offer to sign up for an 'accountability partner' increase or decrease the rate at which people DO XXX activity?

• Does it lead to greater overall XXXlinked donations per visitor over the next 1 year interval?

Generally:

• Does 'social accountability' help to encourage XXX activities and promises and the fulfillment of these? Does the 'fear of being held accountable' discourage people from making commitments?

(Optional: brief on background theory and previous evidence)

Point of contact (at organization running trial)

You can enter more than 1 person here, including an external organizer (like JS Winchell), but ideally, also someone inside the organization.

Add 'academic/research lead' here if there is one

Timing of trial (when will it/did it start and end, if known)

Digital location where project 'lives' (planning, material, data)

The present Gitbook/and a nested Github repo folder could be ideal. Please give a precise link so others could access it.

Environment/context for trial

(Is it on a web page, a google advertisement, a physical mailing, etc)

Participant universe and sample size

• Who will be targeted or who do you expect to be part of the trial?

• (Somewhat optional) How many people (or 'units') do you expect to be involved (median guess)?

• (Optional): How many do you expect will have a 'positive outcome' (e.g., a 'conversion')?

Key treatment(s)

Description, link exact language/content if possible

Treatment assignment procedure

• At what level is it varied? (individual visitors, postal codes, days of the week, etc)

• How are treatments assigned ('blocked randomization', 'adaptive/Thomson sampling', etc.)

  • If you are using a 'set Google, Facebook etc algorithm', just input the settings you used here, and/or link the (Google, FB, etc) explanation

• How many/what shares are assigned to each treatment?

Outcome data

• What measures (outcomes, other features) will be collected?

• When and how

• Where will the data be stored, who will have access

Optional/suggested additions

Planned analysis methods, preregistration link, IRB link, connection to other projects and promotions

Ex-post: Reporting results (brief)

Implementation and data collection

1. Did it go as planned? Any departures? (Timing, randomization, design changes, etc)

2. How much/what data was collected? How many observations?

3. Where is the data stored (also link/adjust the above), who has it, and under what conditions?

Basic results/outcomes

1. "Partners and stakeholders opinions": were they happy with the trial? Did they seem to think it was a success?

2. Simplest statement (e.g., "3% donated in the treatment versus 2.2% in the control, with an average amount raised of $4.3 in the treatment and $3.1 in the control')

3. Preliminary interpretation, with statistical test if possible (e.g., 'google Optimize states an 80% chance that the treatment outperformed the control', a Fisher's exact test yields a p=0.06 that a positive donations was more likely in the treatment than the control)

4. "Full analysis"

   1. Who/what when will it be done?

   2. Link to 'where' it will be done (both the 'follow up the pre-analysis plan, and the full write-up, if applicable)

   3. Possibly: Briefly characterize the overall confusions/state of analysis here (state the date last updated)

5. Feeding synthesis and meta-analysis

   1. Which generalizable questions does this inform?

   2. Is data sharable? Key comparable outcomes?

   3. What other work/trials does this relate to?

   4. State of meta-analysis