💡
EA market testing (public)
  • Introduction/overview
    • Introduction & explanation
    • 👋Meet the team
    • 📕Content overview
    • Progress/goals (early 2023)
      • EAMT progress & results
      • Goals, trajectory, FAQs
  • 🤝Partners, contexts, trials
    • Introduction
    • Giving What We Can
      • Pledge page (options trial)
      • Giving guides - Facebook
      • Message Test (Feb 2022)
      • YouTube Remarketing
    • One For the World (OftW)
      • Pre-giving-tues. email A/B
        • Preregistration: OftW pre-GT
    • The Life You Can Save (TLYCS)
      • Advisor signup (Portland)
    • Fundraisers & impact info.
      • ICRC - quick overview
      • CRS/DV: overview
      • 📖Posts and writings
    • University/city groups
    • Workplaces/orgs
    • Other partners
    • Related/relevant projects/orgs
  • 🪧Marketing & testing: opportunities, tools, tips
    • Testing Contexts: Overview
    • Implementing ads, messages, designs
      • Doing and funding ads
      • Video ads/Best-practice guidelines
      • Facebook
      • Targeted ad on FB, with variations: setup
    • Collecting outcome data
      • Facebook ads interface
        • Pivot tables
      • Google analytics interface
      • Google A/B, optimize interface
      • Reconciling FB/GA reports
      • Survey/marketing platforms
    • Trial reporting template
  • 🎨Research Design, methodology
    • Methods: Overview, resources
    • "Qualitative" design issues
    • Real-world assignment & inference
      • Geographic segmentation/blocked randomization
      • Difference in difference/'Time-based methods'
      • Facebook split-testing issues
    • Simple quant design issues
    • Adaptive design/sampling, reinforcement learning
    • 'Observational' studies: issues
    • Analysis: Statistical approaches
  • 🧮Profiling and segmentation project
    • Introduction, scoping work
    • Existing work/data
      • Surveys/Predicting EA interest
      • Awareness: RP, etc.
      • Kagan and Fitz survey
      • Longtermism attitudes/profiling
      • Animal welfare attitudes: profiling/surveying
      • Other data
    • Fehr/SOEP analysis... followup
      • Followup with Thomas Ptashnik
    • Further approaches in progress
      • Profiling 'existing traffic'
  • 📋(In)effective Altruistic choices: Review of theory and evidence
    • Introduction...
    • The challenge: drivers of effective/ineffective giving
      • How little we know...
    • Models, theories, psych. norms
    • Tools and trials: overview
      • Tools/interventions: principles
      • Outcomes: Effective gift/consider impact)
        • (Effectiveness information and its presentation)
        • (Outcome: Pledge, give substantially (& effectively))
          • (Moral duty (of well-off))
        • Give if you win/ conditional pledge
      • Academic Paper Ideas
  • Appendix
    • How this 'gitbook' works
      • Other tech
    • Literature: animal advocacy messaging
    • Charity ratings, rankings, messages
    • "A large-scale online experiment" (participants-aware)
  • Innovationsinfundraising.org
Powered by GitBook
On this page
  • "Concise reporting template"
  • General idea, main 'hypothesis' (if there is one)
  • Point of contact (at organization running trial)
  • Timing of trial (when will it/did it start and end, if known)
  • Digital location where project 'lives' (planning, material, data)
  • Environment/context for trial
  • Participant universe and sample size
  • Key treatment(s)
  • Treatment assignment procedure
  • Outcome data
  • Optional/suggested additions
  • Ex-post: Reporting results (brief)
  • Implementation and data collection
  • Basic results/outcomes

Was this helpful?

Edit on GitHub
Export as PDF
  1. Marketing & testing: opportunities, tools, tips

Trial reporting template

For each proposed/ongoing/past trial, we should report the following minimal details, with links (proposed template)

PreviousSurvey/marketing platformsNextMethods: Overview, resources

Last updated 2 years ago

Was this helpful?

For each proposed/ongoing/past trial, let's try to report the following minimal details, with links (proposed template) If you don't have time and you have another clear presentation of most of this, please link or embed it.

"Concise reporting template"

Please keep your answers brief -- if you want to give more detail (which is not necessary) please link a later section or external page. _

Short version of this template (link copy-opens a new version for you to work in)

General idea, main 'hypothesis' (if there is one)

Firstly, what is this promotion trying to do (e.g., 'encourage signups for giving pledges')?

But more importantly, what are you trying to learn here... What might you have a better understanding of after the trial than you did before the trial?

E.g.,

Specifically:

  • Does the opportunity we offer to sign up for an 'accountability partner' increase or decrease the rate at which people DO XXX activity?

  • Does it lead to greater overall XXXlinked donations per visitor over the next 1 year interval?

Generally:

  • Does 'social accountability' help to encourage XXX activities and promises and the fulfillment of these? Does the 'fear of being held accountable' discourage people from making commitments?

(Optional: brief on background theory and previous evidence)

Point of contact (at organization running trial)

You can enter more than 1 person here, including an external organizer (like JS Winchell), but ideally, also someone inside the organization.

Add 'academic/research lead' here if there is one

Timing of trial (when will it/did it start and end, if known)

Digital location where project 'lives' (planning, material, data)

The present Gitbook/and a nested Github repo folder could be ideal. Please give a precise link so others could access it.

Environment/context for trial

(Is it on a web page, a google advertisement, a physical mailing, etc)

Participant universe and sample size

  • Who will be targeted or who do you expect to be part of the trial?

  • (Somewhat optional) How many people (or 'units') do you expect to be involved (median guess)?

  • (Optional): How many do you expect will have a 'positive outcome' (e.g., a 'conversion')?

Key treatment(s)

Description, link exact language/content if possible

Treatment assignment procedure

  • At what level is it varied? (individual visitors, postal codes, days of the week, etc)

  • How are treatments assigned ('blocked randomization', 'adaptive/Thomson sampling', etc.)

    • If you are using a 'set Google, Facebook etc algorithm', just input the settings you used here, and/or link the (Google, FB, etc) explanation

  • How many/what shares are assigned to each treatment?

Outcome data

  • What measures (outcomes, other features) will be collected?

  • When and how

  • Where will the data be stored, who will have access

Optional/suggested additions

Planned analysis methods, preregistration link, IRB link, connection to other projects and promotions

Ex-post: Reporting results (brief)

Implementation and data collection

  1. Did it go as planned? Any departures? (Timing, randomization, design changes, etc)

  2. How much/what data was collected? How many observations?

  3. Where is the data stored (also link/adjust the above), who has it, and under what conditions?

Basic results/outcomes

  1. "Partners and stakeholders opinions": were they happy with the trial? Did they seem to think it was a success?

  2. Simplest statement (e.g., "3% donated in the treatment versus 2.2% in the control, with an average amount raised of $4.3 in the treatment and $3.1 in the control')

  3. Preliminary interpretation, with statistical test if possible (e.g., 'google Optimize states an 80% chance that the treatment outperformed the control', a Fisher's exact test yields a p=0.06 that a positive donations was more likely in the treatment than the control)

  4. "Full analysis"

    1. Who/what when will it be done?

    2. Link to 'where' it will be done (both the 'follow up the pre-analysis plan, and the full write-up, if applicable)

    3. Possibly: Briefly characterize the overall confusions/state of analysis here (state the date last updated)

  5. Feeding synthesis and meta-analysis

    1. Which generalizable questions does this inform?

    2. Is data sharable? Key comparable outcomes?

    3. What other work/trials does this relate to?

    4. State of meta-analysis

🪧
as Google Doc HERE