Giving What We Can's mission is to make giving effectively and significantly a cultural norm. GWWC has updated their 2022 strategy. They are looking to significantly increase their marketing activity by producing videos, funding ads, and conducting systematic and robust research. As such there will be a large crossover between our work and theirs. This section highlights our collaborative efforts.

Presentation: overview

Ideas and opportunities

We want to learn from existing work, run tests on the GWWC platform, and support research into this.

Stages of the funnel:

1. Awareness & Consideration

   Increase casual visitors and raise curiosity

2. Conversion & Acquisition

   Donate or pledge to donate

3. Retention

   Fulfill and report pledge

4. Advocacy

   Promoting GWWC to others

Some key questions

• “What should the call to action be for the casual person in the funnel?”

• Testing all parts of funnel/pledge journey; website, welcome messages/welcome packages, reminders and thank-you's

Completed studies: See sections below

Introduction

Our mission, what we are trying to do and why, most recent updates, and the organization of our team

Five Key Sections

1) Partner Organizations and Trials 🤝

In this section, you will find reports of the trials we have run with organizations, including Giving What We Can and One For the World.

2) Marketing & Testing: Opportunities, Tools, Tips 🪧

Here we share tools to implement planned trials, as well as tips relevant to 'doing marketing'. We answer questions like how to set up campaigns and track outcomes on various platforms. See especially "Implementing ads..." and "Collecting data..."

3) Research Design and Methodology 🎨

We discuss qualitative and quantitative research design and methodology issues that are relevant to the trials we are running. Pages in this section will be linked in reports when relevant to a particular trial.

4) Profiling and Segmentation Project 🧮

Our profiling project aims to help better understand what sorts of people are amenable to EA-related ideas and to taking EA-favored actions.

5) (In)effective Altruistic Choices: Review of Theory and Evidence 📋

We've done a review of existing literature: to inform the trials we are running, and to identify important research topics. This includes What is known/models of effective giving and Principles and theories behind potential trials.

You can find references, tech support, and other resources in the appendix.

What is this 'Gitbook' meant for?

The three key aims of this public gitbook are to:

1. Convey who we are, what have accomplished, and the scope of our work to funders, people in the broader EA community, and people not yet involved in the project who would be interested in joining

2. Share tools and knowledge with people in the EA/global priorities community who will apply it to their work. We are building a knowledge base. Content in the public gitbook can inform and support a diverse set of projects (i.e., implementing marketing campaigns, fundraising initiatives, academic research)

3. Seek feedback on our work. This includes technical and industry feedback on implementation and academic expertise (literature reviews and frameworks to consider, methodology, and experimental design).

4. (Grouped by organizational partner.)

5. We include background information on each organization and its priorities for testing.

In the Partner Organizations and Trials section, you will find reports of the trials we have run with organizations, including Giving What We Can and One For the World (OftW).

These trials are also cataloged in our Airtable: Shared view of the relevant trials (that is...); links, categorization provided.

Our primary approach and goals

We want to identify the most effective and scalable strategies for marketing EA and EA-adjacent ideas and actions. To do this, we believe that running real-world marketing trials and experiments with EA-aligned organizations will provide the best evidence to act upon. By systematically varying the messaging, framing, and contexts, we can map out 'what works better where'.

We believe this approach is likely to be the most fruitful because:

1. Using naturally-occurring populations in real-world settings with meaningful costly choices and outcomes will lead to more relevant findings. In comparison to convenience samples of undergraduates or professional survey participants who are aware that they are doing a research study, we anticipate greater:

   • Internal validity: our results are less likely to be influenced by biases, such as acquiescence bias and hypothetical decision-making.

   • External generalizability/environmental relevance: the context we are testing is similar or identical to the context we care about.

2. We will "learn by doing" by encountering unanticipated obstacles and learning about practical implementation issues involved with advertising, promotion, and communication.

3. We can share what we learn with relevant EA organizations and audiences. They then can build on our findings, rather than having to repeatedly make mistakes themselves.

4. The trials themselves should also have a direct positive value in promoting EA.

5. There is limited downside risk. We are generally not testing risky messages and are careful to avoid diluting or misrepresenting EA's core ideas.

Key themes, priorities, and 'high-value questions'

This project primarily aims at:

1. Robust and generalizable insights that improve communication and messaging

2. Meaningful and relevant long-run outcomes, such as;

   • Creating new, strong EAs by getting people more interested and involved in EA ideas, actions, and the community

   • Having people consider and identify with key values and practices, such as making meaningful altruistic choices, considering effectiveness and impact in doing so, strong analytical and epistemic practices, and broad (or carefully considered) moral circles

   • Substantial impactful donation behavior and choices

3. Across a range of EA causes and groups (longtermism, global health, animal welfare)

In the document below (EAMT: HVQs), we consider the shared goals, paths, and questions that are valid across organizations. Specifically, these are actionable and promising themes and projects that can be implemented, measured, and communicated fluidly throughout the EA network.

What is the "EA Market Testing Team"?

We are a small group of researchers and practitioners. This project is organized by David Reinstein , who maintains this wiki/Gitbook and other resources.

We aim to promote awareness and understanding of the core ideas of Effective Altruism (EA), and "to make giving effectively and significantly a cultural norm." We consider marketing campaigns, charitable appeals, events, and public communication, working both with our partner organizations and in independent surveys and trials. We want to improve the design and messaging of organizations like Giving What We Can and 80000 Hours to improve their outreach methods and maximize their impact. Measuring and testing 'what works and when': While helping these organizations do marketing and communication we are also testing and analyzing this rigorously. We help run and track careful data collection and rigorous controlled trials, as well as helping to organize the reporting of less rigorous trials. We robustly analyze the results to better understand which approaches tend to have a more positive impact. Communication: We track, organize, and share what we have learned with the EA community, building and organizing resources and a knowledge base. This will address questions such as:

• How to implement and test marketing campaigns?

• What has worked to promote EA?

• What motivates/blocks effective giving?

• What profiles of people are most likely to be interested in effective altruism?

We strive to be transparent. We want to report and share our data, procedures, code, and evidence without overselling the results.

We believe this is the first organized collaboration of its kind. We aim to...

• Coalesce our understanding and evidence on barriers and facilitators of effective altruism, effective giving, and effective action

• Run a broad set of high-powered trials (large samples, high-stake real-world contexts, substantial differences between conditions)

  • ... to gather evidence on what works best to promote meaningful actions in specific cases,

  • ... while aiming at generalizable principles and approaches

• Do profiling, survey, and segmentation activities and trials, building evidence on 'which types of people' are most responsive to effective giving messages and appeals

• Share our results, data, and tools, with the relevant EA and research-interested communities. This will enable more and better outreach, promotion, testing, and insight.

What is our mission? Updating...

As EAMT has progressed, we have encouraged others to do work and pursue initiatives in the 'space' of studying EA messaging, and marketing EA and effective giving. We hope that the resources we have provided, and the connections we have made have contributed to this. As the space changes, the EAMT mission, scope, and activities are adjusting as well.

We are moving towards a heightened focus on

1. Advising, proposing, and helping to design and coordinate experiments, trials, and initiatives.

2. Transparent presentation of the results, rigorous statistical analysis

3. Synthesizing, sharing, and communicating this knowledge and skills base

This work provides substantial public goods, whose benefit is shared among the partner organizations and the EA community.

Other relevant/new organizations and initiatives

• #other-marketing-implementation-resources (including User Friendly, Good Impressions, and Altruistic Agency)

• Rhetorical.org

• Giving What We Can 'Bequests' (a project we have encouraged and advised)

• Effective Altruism Psychology Lab (NYU)

• Effective Giving Collaboration and Summit

We believe the EA Market Testing Team is the first organized collaboration of its kind.

Goals and FAQ

Goals/FAQ link (below in detail, scroll outside margin to skip past it)

What have we accomplished?

For an overview of our progress and ongoing work, see the 'progress and results' document we are building. (Below in detail, scroll outside margin to skip past it)

Note that we cannot publicly share details of ongoing and upcoming trials. We aim to share the results when it is possible. We aim to integrate shareable aspects of this private doc.

"Testimonials"

How to get involved?

If you are interested in getting involved with our project or have feedback for us, contact David Reinstein at daaronr AT gmail.com.

Next, check out the Gitbook content overview.

(For an explanation of this Gitbook's structure, content, and aims.

1. This quote comes from the 2022 Giving What We Can strategy document.

2. However, we are also careful to be efficient, recognizing the tradeoffs between rigorous experimental design and practical marketing.

3. Including testing templates, guides and implementation tips

Summary

Main Question: Do some message themes work better than others for drawing visitors to Giving What We Can’s landing page?

Main findings: 'Social proof messages' on Facebook ads were most effective at generating landing page views per dollar compared to other message themes (effectiveness, services, giving more, and values).

Future directions: There were significant differences in 'link clicks per dollar' on the different messages by age. We recommend a systematic test to determine if age makes a difference in the relative effectiveness of social proof and values messages. Future studies could explore why the social proof message was more effective in this study than the previous giving guide study and the importance of the message to “join” the movement as social proof.

Possible connection between this trial and the : Note that the two best-performing messages both prompted the user to “join” a movement or a group of people (perhaps an elite group); but beware .

to report below.

Pre-trial reporting template

General idea, main 'hypothesis' (if there is one)

In this test, we are aiming to find out if one 'theme' of messages resonates better with our target audience than others.

If we knew which 'themes' were most effective with our advertising, then we could create more ads on this theme and improve our conversion.

Specifically, which of the following themes resonate with our target audience the most:

• effectiveness

• giving more

• social proof

• values

• services

On choosing an objective of this test, originally I planned to use link clicks, but this is not the most high quality indicator of conversion, and when I tried to use newsletter signups Facebook warned me that I might not see any conversions at all... So instead, the campaign will optimise for landing page views, which is slightly better than a link click and will generate enough conversions that we should [see?] we statistically significant results.

Point of contact (at organization running trial)

Grace Adams

Timing of trial (when will it/did it start and end, if known)

Trial will run for 7 days on GWWC's ad account, from 9.30am AEDT Friday 25 Feb to 9.30am AEDT Friday 4 Mar.

Digital location where project 'lives' (planning, material, data)

Environment/context for trial

This test will take place on Meta platforms including Facebook and Instagram

Participant universe and sample size

• We are targeting a "Philanthropy and Lookalikes (18-39)" audiences, based in UK, US or Netherland

• Estimates from Facebook: Reach is expected to be 1.4K-4.1K per day (7 days) per ad set (5 ad sets) = 49K-143K

• Estimates from Facebook: Conversion is expected to be 10-30 landing page views per day (7 days) per ad set (5 ad sets) = 350-1050

Key treatment(s)

We are using the GWWC Brand Video by Hypercube as the creative across all tests. Although it did not perform as well as our other ads in the Giving Guide campaign, I think that it will interfere less with our messages we aim to test.

Mock up of ad:

Treatment assignment procedure

• This test has been set up as an A/B test through Facebook, testing each campaign head to head, each campaign covers one theme, with the different ads as a child.

• This will allow us to test which theme was better, not just which individual ads

• A/B testing on facebook will ensure that the audiences fall into an individual treatment group

Outcome data

• Primary measure will be cost per landing page view, but secondary measures such as CPC, 3 second video plays, email sign ups will also be tracked

• Data will live on Meta ads platform

See also the cross-organization (=placeholder for now) and the tips on

YouTube Remarketing

July 20, 2021: GWWC launched a YouTube remarketing campaign. That means that when someone goes to the GWWC website, leaves, and then goes to YouTube we show them one of the following videos:

Algorithm decides which video to present to people.

Understanding assignment, proposing experimental design ’s questions:

Q: Is each video assigned to a different situation or are videos randomly chosen to be displayed? If the latter, you could randomize videos by location and see if the different videos were more or less effective. Alternatively, just randomizing the whole campaign seems like a good idea to me....

A: Videos are selected based on the likelihood of the user watching >30 seconds (by the algorithm) ... randomization by individual will be hard because users don't click and act right away. Instead I think we have to randomize by geography

Results summary (Early, JS Winchell; may need update)

Most important takeaway: It costs $1 to get a website visitor to watch 1h of your videos! High level metrics

• Cost: $205

• Views: 6,071 (a view is when a user chooses to watch >30s of an ad)

• Total watch time: 223 hours (~$1/h)

• Unique viewers: 4,937 (this is an estimate)

• Average impressions per user: 5.8

• View rate: 20% (20% of the time people choose to watch more than 30s)

• CTR: 0.37%

• Average CPC: $1.83

• Conversions (users spending >30s on the website): 2

• Thinking: 'This is not a good tactic for driving site traffic or donations (although we could optimize for this instead if we wanted)'

Interesting observations\

1. Efficiency has significantly improved over 3 weeks

• Cost per view has gone down from $0.05 per >30s view to $0.02 per >30s view

• Views have increased 75% without increasing budget (from 220/day to start to 386 yesterday)

• You can see this data by video if you are interested to control for video length

3. Your best video had a view rate (% of time people choose to watch >30s) twice as good as your worst video 4. You can see view rate by age, gender, and device in the "Analytics" tab

Possible next steps

• Could add "similar audiences" which is when we let Google use machine learning to find people similar to your website visitors and also show ads to them

• Could walk David Reinstein and Joshua Lewis through the UI so they can get a sense of the metrics/reporting available and how it could be used for research

Along with GWWC, we tested marketing and messaging themes on Facebook in their Facebook Lead campaigns. Across four trials we compared the effectiveness of different types of (1) messages, (2) videos, and (3) targeted audiences.

A summary of this has been shared as a on the EA Forum:

We build the results and analysis transparently in the

Context: Facebook ads on a range of audiences

Also informative about costs and the 'value of targeting different groups' in this context.

Key findings:

• “Only 3% of people give effectively,” seems to be an effective message for generating link clicks and email addresses, relative to the other messages.

• Lookalike and animal rights audiences seem to be the most promising audiences.

• Demographics are not very predictive on a per-$ basis.

Key caveats

The outcome is 'click to download the giving guide'.

We are a group of researchers and practitioners across a range of fields (Economics, Psychology, Marketing, Statistics) and organizations, particularly those interested in effective charitable giving and effective altruism. This is outlined in the , with embedded views below.

This project is organized by , who maintains this wiki/Gitbook and other resources.

As individuals and organizations, we are goal-driven and impact-driven: we are in this to improve the world, particularly through directing funds and support to the most effective rercauses and interventions. Because we share these common goals, we are better aligned for collaboration than typical academics and charitable organizations. We have an unprecedented opportunity to collaborate, learn what works, and 'move the needle'.

We are actively collaborating with the following organizations (links indicate publicly reportable trials)

The presentation of options on GWWC's '' were randomly varied at the individual browser level over a certain period to see which option increased pledges.

A summary of this has been shared as a on the EA Forum.

Summary of trial and results

Giving What We Can (GWWC) has three giving pledge options, displayed in the 'Original presentation version' below.

From April-July 2021 they ran a trial presenting its 'pledge page' options in three slightly different ways. Considering 'clicks on any button' as the outcome, and a Bayesian 'preponderance of evidence' standard...

• "Separate Bullets for Other Pledges" was the most successful presentation. It only showed a box for "The Pledge", with the other options given in less prominent bullet points below. This had about a 20% higher incidence rate than the Original presentation.

• "Pledge before Try Giving" was the least successful presentation this was like the one displayed above, but with "Try Giving" in the central position. This had about a 23% lower incidence rate than the Original presentation.

General idea and main hypothesis

GWWC: How can we present pledge options to maximize positive outcomes (pledges, fulfillment)?

General: For those considering making substantial giving pledges (of a share of their income), how does the presentation of these 'pledge options' matter?

Theories and mechanisms to consider:

• Too many options may lead to 'indecision paralysis'

• The signaling power of choice; e.g., if there's a 'more virtuous choice' I may feel that my 'middle choice' looks less good by comparison

Background and context

1. "Try Giving" (1% of income),

2. "The Pledge" (10% of income)

3. The "Further Pledge" (donate all income above a living allowance).

Three versions of this page were randomly presented (between 19-21 April and 10 July 2021)

The content of the key 'choice button' part varied between these three versions

1. "Original:" A block of three (in the order of commitment) 'The Pledge' (10%) in the center and highlighted (see above)

2. "Pledge before TryGiving": A block of 3 with "Try Giving" (1%) in the center and highlighted

3. "Separate Bullets for Other Pledges": A single block for 'The Pledge' (10%), with the other pledges given as clickable bullet points below (as well as a bullet for the 'company pledge' ... which had a different presentation in other versions)

The version presented stayed constant according to an individual's IP cookie tracking.

Participant universe and sample size

• 'Everyone going to the above page' within the above time duration.

• People interested in GWWC pledges'

Sample size: see below, from Google Analytics

Key treatment(s)

1. "Original" (Block of 3 in order of commitment, Middle Pledge in Center)

2. "Pledge before TryGiving" ... as above but with Try Giving and The Pledge swapped, and Try Giving (in the center) highlighted

3. "Separate Bullets for Other Pledges" (see below)

Treatment assignment procedure

• Three versions of this page were randomly presented

Outcome data

Statistics on Google Analytics: This records only 'pressed any button' (any pledge) as the successful outcome.

Ex-post: Reporting results (brief)

Implementation and data collection

1. From shared image from Google Analytics:

'Experiment sessions' (observations) by treatment (as labeled on Google Analytics shared image):

Original: 2588

Pledge before Try Giving: 2686

Separate Bullets for Other Pledges: 2718

Total: 7992 sessions (=2588+2686+2718)

3. Where is the data stored ... [noted above]

Basic results/outcomes

Quick interpretation

The "separate bullets for other pledges" seems to have been the most successful, with an 0.49% higher (percentage point) incidence rate than the 'Original', i.e., a 22% higher rate of pledging (2.69 vs 2.20).

These differences seem unlikely to be statistically significant in a conventional sense. Still, Google analytics (presumably a reasonable Bayesian) model states an 80% chance that this is the best treatment, and this seems useful and informative.

The "Pledge Before Try giving" treatment performed substantially worse than the original.

These differences are meaningful–consider the 'posteriors':

Downloading the 'Analytics data' behind the above graphs, we see:

This suggests it is very reasonable to think that 'Separate Bullets' is substantially better

• a 2.5% chance that 'Separate Bullets' (SB) has an 18% (or more) lower conversion rate than 'Original'

• a 22.5% chance on SB being between 18% worse and 4% better

• a 25% chance of SB being 4-20% better

• a 25% chance of SB being 20-36% better

• A 22.5% chance of SB being 36-76% better

• A 2.5% chance of SB being more than 76% better

We can also combine intervals, to make statements like ...

• a 50% chance of being 4-36% better

• a 50% chance of being 20-76% better

For 'Pledge before...' (PB) we can state, e.g.,

• PB has a 75% chance of being at least 11% worse than Original

• and a 50% chance of being at least 23% worse than Original

Intuitive interpretation

Perhaps giving people more options makes them indecisive. They may be particularly reluctant to choose a “relatively ambitious giving pledge” if a less ambitious option is highlighted.

This could also involve issues of self and social signaling. If the 'main thing' to do is a 10% pledge (as in "separate bullets"), then this may seem a straightforward way of conveying 'I am generous'. On the other hand, if the 'Further pledge' is fairly prominent, perhaps the signal feels less positive. And if the '1% pledge' is made central, 10% might seem more than a necessary signal.

The "pledge before try giving" may perform the worst because it makes the 'Try Giving' pledge a particularly salient alternative option. (In contrast, the "Original" at least makes 'The 10% Pledge' the central and the middle option.)

Caveats

I am assuming that the 'outcome being measured here' is whether the person 'clicked on any giving pledge'; this is what Luke has conveyed to me

I assume this is 'conversions ever from this IP', and 'sessions' represents 'how many different IPs came to the treatment'. If it's something else (e.g., each 'session' is a 'visit' from an individual), this could reflect these people converting in fewer sessions but not necessarily being more likely to convert overall. Even if this is 'by IP' the alternative interpretation 'not converting now but maybe later' may still have some weight if people are entering through multiple devices.

Background on OftW

• OftW has a donor base of ~700 active donors, ~1650 pledged donors (who pledged but haven't started donating yet) and ~2000 lapsed donors.

• 80% (of donors?) are in the USA

• Focus on global health charities

• They focus on donations to GiveWell charities ... but technically OftW pledgers can give to any 510c3

Some key goals

Reinstein/Lewars conversation notes

• Activating more donors who took the pledge at university, so their donations actually start;

• Retaining donors for longer once they activate;

• Upselling donors to give more over time (either more as a raw amount, e.g. 'keeping pace' at 1% of their income; or more as a percentage, e.g. 'graduating' to take the 10% GWWC pledge)

• Acquiring new donors with fewer touchpoints, e.g. via online advertising, via our website etc. (we currently get ~0 organic sign-ups)

Chloe's OKRs Notion

Who/what/how to test, learn, and adapt

Pipeline/groups/segments

1. Pledgers

2. Active donors, i.e., "Activated pledgers" (Chloe is thinking of segments to this and how to appeal to them)

   1. Second tier -- people who have given each month for 12+ months; "Legacy donors" (DR: maybe 1x per year high-value donors should be in this group)

3. One-time donors (these may or may not be pledgers)

4. Cancelled

5. Payment failures

• Another group worth considering: 'pledge-curious supporters'

Goals/actions

• 'Activating' Pledgers as donors (pledged but not donated)

• Active donors

  • Retain

  • Upsell (maybe only to the second tier?)

• Acquiring pledges, perhaps from a 'pledge-curious group'

Interested in knowing more about

• Content -- expand our ability to tell stories about the beneficiaries

  • Ways to tell these stories

• Frequency (of comms with supporters)

Communications contexts

• Platforms: Social media, email flows

• Telling stories in a corporate context

Typical audiences have been students and young professionals, but there is interest in corporate outreach

• Zoom and lunchtime talks in corporate contexts (How many? Seems very promising!)

  • How many people are activating/pledging following these lunch+learn?

Typical donor journey:

We are in the process of creating these homepages and setting up conversion tracking. As OFTW has ~0 organic sign ups currently, we are testing for a variety of conversion routes, including: [Todo: clarify this]

• university campus, someone I like tells me they are involved in OftW, asks me to come along with free food

• at some point I take the pledge

• It is not a highly controlled process

• asking us (staff) a question by email

• joining a group call with others wanting to learn about effective donating (Kennan as dir. of chapter management)

• taking the pledge

• making a one-off donation

Some rough numbers

650 active donors

1500 people in pipeline (pre-activation date)

750 new people a year are recruited... thinks it would be 2-2.5k

OFTW has a donor base of ~700 active donors,

~1650 pledged donors (who pledged but haven't started donating yet) and

~2000 lapsed donors.

Ongoing/completed/upcoming experiments

Email upsell emotion/impact message trial (see below)

University experiment - redacted as being prepared

• Homepage message testing

• Activation trial

Academic-linked authors: David Reinstein, Josh Lewis, potentially others going forward

Implementation and management: Chloe Cudaback, Jack Lewars

AsPredicted questions

1) Have any data been collected for this study already?

No, no data have been collected for this study yet.

2) What's the main question being asked or hypothesis being tested in this study?

Are effectiveness-minded (EA-adjacent) donors and pledgers more motivated to donate by

1. "A": A (non-quantitative) mention of impact and effectiveness (in line with the standard OftW pitch)

2. "B": Emotional appeals and 'identified victim' images

Framing this in terms of the psychology, social science, and philanthropy literature:

"Does the Identifiable Victims Effect (see e.g., meta-analysis by Lee and Feeley, 2016) also motivate the most analytical and committed donors?"

3) Describe the key dependent variable(s) specifying how they will be measured.

• d_don_specific: Whether the person receiving the series of emails makes an additional 'one time gift' following the link at OftW, within the OftW interface, during the 'Giving Season', a time-period that (for this preregistration) we declare to begin on receipt of this first email and end on 15 January 2022.

• don_specific: The total amount donated through the above

• don_general_gs: (If observable), the amount the person donates during the 'Giving Season', as observed through the OftW/donational/Plaid network

• don_general_1yr: (If observable), the amount the person donates during the 'Giving Season' and for the following year (ending 15 January 2023) as observed through the OftW/donational/Plaid network

• d_continue_pledge_1yr: Whether the person is still an active OftW pledger a year after the current giving season (15 January 2023)

4) How many and which conditions will participants be assigned to?

Two conditions (treatments):

A. "Impact"

B. "Story/Emotion"

Assignment details

Participants (c 4000 people at various points in the One for the World pledge process) will be split into groups (blocks) by previous donation behavior or point in the process. (OftW have mentioned, pledgers still in school, active donors, and lapsed donors).

Within each group, they will be randomized (selection without replacement to ensure close-to-exact shares) into equal shares in treatments A and B.

Treatment specifics (i.e., 'experimental conditions')

A series of three emails will be sent, with participants remaining in the same treatment across all three emails.

See actual texts for design and timing HERE

Example content differences, from email 1:

A. Impact version:

As of 2021, One for the World has had a tremendous impact on the lives of those that are helped by our charity Top Picks programs:

[IMPACT SINCE 2021 GRAPHIC]

B. Story/Emotion version:

Here’s our first story this season from Eunice of Kenya. When asked how her life changed when she received the first cash transfer from our partner organization, GiveDirectly, she responded”

“I have been able to make new goals and achieve them since I started receiving this money [from GiveDirectly]. I have been able to buy a piece of land that would have taken [me] many years to earn [enough to buy the land]. I was also able to buy livestock, like goats. I have even managed to dress my family properly by buying them decent clothing. Lastly, I have even been able to [pay my children’s] school fees without any strain.” (Source GiveDirectlyLive)

[PICTURE OF EUNICE]

5) Specify exactly which analyses you will conduct to examine the main question/hypothesis.

We will report all of the following analyses, with our preferred method in bold:

Binary outcomes:

• Fisher's exact test

• Bayesian Test of Difference in Proportions (as in here), with an informative beta distribution for the prior over the incidence rate in each treatment, with a parameter based on the incidence rates for similar campaigns in the prior 2 years.

Continuous outcomes:

• Standard rank-sum tests (Mann–Whitney U test)

• Simulation/permutation based tests for whether the mean (including 0's) is higher in group A or B (including 0's)

• ... same for median, but medians will almost always be 0, we anticipate

• T-test with unequal variance

All tests will be 2-sided.

We will also report Bayesian credible intervals and other Bayesian measures for the proportion tests. We may also explore Bayesian approaches for the continuous outcomes, e.g., Bayesian beta regression.

We also anticipate reporting multiple-hypothesis-test corrections, but we are not pre-registering a method. Our approach to this is likely to follow that of List et al (2017), which this paper applied to a similar domain (charitable giving experiments with multiple donation-related outcomes).

We will report confidence intervals on our results as well as Bayesian credible intervals under flat and weakly informative priors. Where we have a 'near-zero' result, we will try to put reasonable bounds on it to convey the extent of our certainty that the true effect or parameter was fairly small.

Where situations arise that have not been anticipated in our preregistration and pre-analysis plan, we will try to follow the Don Green lab standard operating procedures unless there is a very strong reason to deviate from this, which we will specify.

6) Describe exactly how outliers will be defined and handled, and your precise rule(s) for excluding observations.

• Included: All individuals who received this mailing.

\

We will not exclude any observations from the sample, unless they make it clear to us that they are aware of this trial.

We will not Windsorise or exclude outliers.

7) How many observations will be collected or what will determine sample size?

A series of three campaign emails will be sent out by OftW to their regular email lists, to roughly 4000 participants, as described above

Targeted dates: November 10, November 18, November 23, all in 2021, but these may be delayed for feasibility

Other

Anything else you would like to pre-register? (e.g., secondary analyses, variables collected for exploratory purposes, unusual analyses planned?)

Exploratory and secondary hypotheses/questions/analyses

Secondary hypotheses and questions

Which treatment motivates a higher rate of...

• Email open rates (note, as we have three obs per participant, we will need random effects or clustered standard errors). and

• Use click rates (with same caveat)?

We consider these as secondary because the click and open rates do not necessarily strongly relate to outcomes of interest, particular among this set of already effectiveness-minded donors. These outcomes may simply reflect attention or curiosity about the content.

\

Exploratory: what factors (especially gender, university/student status, university subject) predict which treatment leads to greater donation (incidence and amount)

Note that our partner is planning to use this trial to inform future trials and experiments, particular for the 'Giving Tuesday' season itself.

Power calculations

We did not have time to do even simple power calculations before the start date of this experiment. However, we will try to conduct these before we obtain any of the data, and update this preregistration.

Context: Donation 'upsell' to existing pledgers

Question: Are effectiveness-minded (EA-adjacent) donors and pledgers more motivated to donate by

1. "A": (non-quantitative) presentation of impact and effectiveness (as in standard OftW pitch)

2. "B": Emotional appeals and 'identified victim' images

General idea, main 'hypothesis'

Are effectiveness-minded (EA-adjacent) donors and pledgers more motivated to donate by

1. "A": (non-quantitative) presentation of impact and effectiveness (as in standard OftW pitch)

2. "B": Emotional appeals and 'identified victim' images

In the context of One for The World's (OFTW) 'giving season upselling campaign', potentially generalizable to other contexts.

Academic framing: "Does the Identifiable Victims Effect (see e.g., the meta-analysis by Lee and Feeley, 2016) also motivate the most analytical and committed donors?"

Background and context

One for The World's (OFTW) 'giving season upselling campaign''

10 emails total over the course of November were sent in preparation for GivingTuesday

Point of contact (at organization running trial)

Timing of trial

Targeted dates: November 10, 18, 23, all in 2021, but may be delayed for feasibility

Digital location where project 'lives' (planning, material, data)

Present Gitbook, Google doc linked below, preregistration (OSF), and github/git repo

Environment/context for trial

Emails ... to existing OftW pledgers (asking for additional donations in Giving Season)

All 10 emails had the same CTA: make an additional $100 donation for the giving season/GivingTuesday on top of their recurring monthly pledge donation.

Participant universe and sample size

Roughly 4000 participants, as described.

A series of three campaign emails will be sent out by OftW to their regular email lists, to roughly 4000 participants, as described.

Key treatment(s)

Basically:

• A list of ~4500 contacts (activated pledgers) was split into two treatment groups.

• Treatment Group A received emails that were focused on the contact's impact

• while Treatment Group B received emails that were focused on individual stories of beneficiaries

See preregistration, treatment specifics

Treatment assignment procedure

See preregistration How many ... conditions

Outcome data

Targeting: Donation incidence and amount in the relevant 'giving season' and over the next year, specifically described in prereg under

key dependent variable

Data storage/form:

• MailChimp data (Chloe is sharing this),

• Reports on donations (Kennan is gathering this)

Optional/suggested additions

Planned analysis methods, preregistration link here

Cost of running trial/promotion: Time costs only (as far as I know)

Proposed/implementing design (language)

(Link)

Pre-registration work

Pre-registered on OSF in 'AsPredicted' format, content incorporated here here

Preliminary results

Overview:

The Emotion treatment leads to significantly fewer people opening emails, but more people clicking on the in-email donation link (relative to the standard Impact information treatment). However, we are statistically underpowered to detect a difference in actual donations. More evidence is needed.

Chloe: those emails that appealed to emotional storytelling performed better (higher in-email click rate) than those that were impact-focused.

DR, update: I confirm that this is indeed the case, and this is statistically significant in further analysis.

Evidence on donations

(preliminary; we are awaiting further donations in the giving season) ...

Treatment 1 (Impact): We record

• 1405 unique emails listed as opening a ‘control’ treatment email

• 29 members clicking on the donation link in an email at least once (2.1% of openers)

• 15 members making some one-time donation in this period (about 0.11% of openers, 0.075% of total)

• 8 members emails donating (likely) through the link (0.057%/0.04%)

Treatment 2 (Emotional storytelling):

• 1190 unique emails listed as opening an email (a significantly lower 'open rate', assuming the same shares of members were sent each set of treatment email)

• 56 members clicking on the donation link in an email at least once (4.7% of openers)

• 11 members making some one-time donation in this period (about 0.9% of openers, about 0.055% of total)

• 9 unique emails donating (likely) through the link (0.08%/0.045%)

‘Initial impressions of preliminary outcomes’

• The conversion rates are rather low (0.5%) … but maybe high enough to justify sending these emails? I’m not sure.

• While people are more likely to O_pen_ at least one Impact email, they are more likely to Click to donate at least once if assigned the Emotion email

• But we can't say much for actual donations.

• Given the low conversion rates we don’t have too much power to rule out ‘proportionally large’ differences in conversion rates (or average amounts raised) between treatments …

The figure above seems like a good summary of the ‘results so far’ on ‘what we can infer about relative incidence rates’, presuming I understand the situation correctly …I plot Y-axis: ’how likely would a difference in donations ‘as small or smaller in magnitude’” than we see in the data between the incidence … against X-axis: if the “true difference in incidence rates” were of these magnitudes

Implementation and management: Chloe Cudaback, Jack Lewars

• Our data is consistent with ‘no difference’ (of course) … but it's also consistent with ‘a fairly large difference in incidence’

• E.g., even if one treatment truly lead to ‘twice as many donations as the other’, we still have a 33% chance or so of seeing a difference as small as the one we see

• We can reasonably ‘rule out’ differences of maybe 2.5x or greater

• Main point: given the rareness of donations in this context, our sample size doesn’t let us make very strong conclusions in either direction about donations

See also

We recoded and augmented this analysis within the EAMT Analysis web-book here. More work could be done, if warranted.

Summary of trial and results

In December 2021, TLYCS ran a YouTube advertising campaign in Portland Oregon, involving ‘donation advice’. The top 10% household-income households were targeted with (one of) three categories of videos. One of the ultimate goals was to get households to sign up for a 'concierge' personal donor advising service

Quick takeaways

• There were very few signups for the concierge advising service. (About 16 in December 2021 , only 1 from Portland.)

• We consider a 'difference in difference', to compare the year-on-year changes in visits to TLYCS during this period for Portland vs other comparison cities.

• This comparison yields a 'middle estimate cost' of $37.7 per additional visitor to the site. This seems relatively expensive. We could look into this further to build a more careful model and consider statistical bounds, if such work was warranted.

General idea and main questions

Specific goal of TLYCS promotion: To get people to click on the ad and go to the 'landing page' of TLYCS. Here, they will fill out a form to request an appointment with a donation advisor. We will simultaneously be raising awareness for TLYCS.

General questions:

1. Can we get people to sign up for donation advice using videos in YouTube Ads?

   • How many sign-up and what sorts of people?

2. Do these ads boost engagement with TLYCS in net? (E.g. donations, website activity, book downloads)

   • "Lift test" on Portland market (analyze with difference-in-difference relative to other markets)

3. Which ads are best at this? (These ads differ in substance as well as in style)

Background and context

Participant universe and sample size

Location: Portland, OR

Audience: Top 10% of household income

People living in Portland, Oregon in the top 10% of household income (approximated by Google) will get an in-stream ad (ad plays before video user intended to watch)

Key treatment(s)

Exposure to a sequence of nine versions of YouTube ad videos. Frequency cap: 6/weeks

Three main 'theme/header' variations (similar, slightly different phrasings)

these variations were crossed with...

Three categories of videos within each theme:

1. "Bravery": Charlie Bresler explains how 'you can save lives without being brave' with small amounts of money for bednets, nutrient micro-doses, etc.

2. $10: Man giving out money to poverty-stricken people in Capetown. Text narrative overlaid describes that $5 can buy a slice of pizza, or an interocular lens to treat cataracts, etc. Leans towards 'identified victims/recipients'.

3. "I want to do good": Colorful puppets sing about giving and donating to save lives. Counters common arguments about 'breeding dependency', fear of administrative waste, etc.

These are organized and linked here.

Each video comes with a site-link extension with a Call to Action:

Treatment assignment procedure

We assigned the particular video treatments to audiences using a YouTube/Google optimization algorithm. This chose videos to maximize the probability that a user chose 'Speak to an Advisor' and filled out the linked form.

Outcome data

• How long people watched the videos for

• Whether they 'clicked through'

• Whether they filled out the form for advising (Algorithm is serving to optimize this)

Results (simple analysis)

Cost per user (first-pass)

A first pass and upper bound on impact and (lower bound on) cost/session

Assumptions/data interpretations

1. The numbers used in our data come from meaningful sessions from unique users

2. The 'date range' is the relevant one for being affected by the advertisements of interest

3. The 'comparison cities' are approximately randomly selected

Most optimistic (unrealistic) bound

Guiding assumption: a counterfactual 0 visits from Portland in this season

• 306 Portland Users (389 Portland site visits) in relevant 2021 period.

• If these were all driven by the advertisement (and counterfactual was 0 visits), this is +306 Users and +389 visits

• Cost $4k

• -->Lower bound on cost of $13.07 per user ($10.28 per visit)

Year-on-Year (maybe reasonable) optimistic bound

Guiding assumption: a counterfactual 'sam as last year' in Portland

• 306 Portland Users (389 Portland site visits) in relevant 2021 period.

• 144 Portland Users (189 Portland site visits) in relevant 2020 period.

• --> 306 - 144 =162 users uptick,

  • (389 - 189 = 200 visits uptick)

• --> $4k/162 = $24.69 Lower bound on cost per user

  • ($4k/200 = $20 per visit)

Difference in Differences comparison to other cities

Guiding assumptions:

• The cities used are fairly representative

• 'Uptick as a percentage' is unrelated to city size/visits last year

• All the cities in the comparison group are 'informative to the counterfactual' in proportion to their total number of sessions

This yields

• 112.5% visits uptick (Year on Year) for Portland in 2020

For all North American cities other than Portland (with greater than 250 000 people):

The average is 46.5 users in the 2020 period and 64.5 users in the 2021 period, an uptick of about 38.8%. This is very similar to the result if we look at all cities which has an uptick of 43.1%.

• 38.8% uptick multiplied by 144 users = 55.9 (‘counterfactual uptick’ in users for Portland)

• 162 - 55.9 = 106 (uptick relative to counterfactual)

• USD 4000 /106 = 37.7 USD cost per additional user through this ad

Note this is a midpoint estimate, we have not yet given statistical bounds.

In the graph below (pasted from the Quarto here), we show these year-on-year upticks in context.

Other outcomes

There were very few signups for the concierge advising service. Only about 16 in December 2021 globally, only 1 of which was from Portland.

Notes

Other detailed notes are in our private Gitbook. More formal and detailed analysis could be done if it seems merited.

We have completed various trials in conjunction with The Life You Can Save, the most recent being theAdvisor signup (Portland) city-level YouTube test. There are a number of additional proposed trials and tests, however, at the moment these considerations are limited to the private Gitbook.

Note that in the past TLYCS has worked with the Graduate Policy Workshop School of Public and International Affairs at Princeton University, who produced the 'Behavioral Insights to End Global Poverty' report embedded below.

https://app.gitbook.com/u/WrM9GjKWCyRyoIjCKt7f0ddJwCr1's research (along with others) considers 'how do potential donors respond to (different presentations of) impact information'. Reinstein and his academic partners ran several experiments, working with (and on) mainstream charities and fundraising platforms.

See work:

and discussion:

Other work is ongoing and cannot be publicly shared yet (see private gitbook if you have access).

See public 'open science' work in progress and preliminary results

ICRC: quick overview

April 2021 mailing addressed to ICRC's existing donor base of:

• active (regular donors),

• warm (last donation between 12+ and 24 months ago) and

• sleepy (last donation 24+month ago) donors.

• 169,919 donors (active donors (58,330; 34,14%); warm donors (48,672; 28.49%); sleepy donors (62,758; 36.73%))

• Mailing goes out to donors in Switzerland (all parts: German, French and Italian)

What's been/being done, what do we know?

Discussion space

See public 'open science' work in progress and preliminary results

• 'Thanksgiving email' trial run in 2 subsequent years

• Super-overoptimistic information (2018), Moderately overoptimistic information (2019)

EA Forum posts

EA seeks to amplify its impact through movement-building. Organizations like 80,000 Hours and CEA are into developing and expanding the EA community. Building EA groups has been at the core of this agenda, especially in elite and influential places (such as top universities). Key aims include 'creating highly engaged EAs' and encouraging people to pursue impactful careers.

Our collaborative goals

Currently, university EA groups operate in conjunction with the Centre for Effective Altruism, but with high levels of autonomy. There is only limited collaboration between groups. Such collaboration could allow them to achieve economies of scale and scope, run more systematic and powerful trials, and share insights and methods that increase student engagement.

The EAMT hopes to help coordinate this, consolidate the evidence, and provide accessible tools to newly-formed groups. We want to help avoid repeating errors and 'reinventing the wheel' each time.

EA groups at universities

The efforts and experience of individual EA groups can provide contextual evidence and insights. The EAMT aims to aggregate this knowledge, find generalizable principles, and disseminate this to the wider EA community. We are focused on meaningful medium-term outcomes, e.g.:

1. Membership and participation in EA organizations, and markers of post-university involvement

2. How career plans are impacted (focusing on particular programs and paths)

3. How research and discourse at universities can be influenced

Relevant organizations and programs

The programs below also aim for generalizable principles; e.g., their 'starter toolkits' are implemented across a range of cities, universities, and settings.

Centre for Effective Altruism

University Group Accelerator Program

Community Building Grants Program

EA Group Organizers Survey

1. The growth and composition of EA groups and their activities

2. The opinions of the group's status from the organizer's point of view

The first component gives insight into priorities and progress. The second can help guide our research and provide insight into the tools required by group organizers to increase group interaction and outreach.

Open Philanthropy

80,000 Hours

80,000 Hours is actively targeting university students and offering them guidance on high-impact career paths. (see private Gitbook, if you have access)

Further outreach

There are some further initiatives in this area but most of the material cannot be shared at the moment (see private Gitbook).

Independent group testing and coordination

In this section, we are putting together documents, trials, and knowledge currently being gathered by different EA groups. As we increase our collaboration with these groups, these trials, ideas and documents will become integrated with the Gitbook and EAMT's work, forming a basis for future work and testing.

Funnel Map

This is our basic understanding of the processes used to draw in new members to EA university groups and fellowships, and how members progress through different stages of engagement. Each stage gives us grounds for testing through the different variations of these approaches. This is not just about testing which methods work for attracting the highest number of new members (i.e., which 'call to action' to use at activity fairs, etc), but also increasing engagement and developing high-level EAs (i.e., fellowship program alternatives, discussion group topics, etc).

Stanford University

Awaiting response from Stanford EA.

University of Chicago

Currently limited to private Gitbook.

MIT, MIT Alignment org

Currently limited to private Gitbook.

EA Israel

Useful findings will be synthesised and integrated here in the future.

Independent University Group Outreach

1. Thus, we hope our efforts will be valuable to these initiatives and groups, by providing and sharing evidence on successful approaches to increasing engagement.

2. Note that the survey does not collect data from the group's *members*, although they do ask about the overall numbers of people who engaged with each group.

• (in schools): trials in preparation, extensive consultation

• 80000 hours: trials in progress, preparation, and analysis; some work joint with Rethink Priorities. Note: we have limited permission to report on these trials

• The Life You Can Save: Trials run and in preparation, limited permission to share

• (HIP): Advising on surveys and approaches

• GiveWell (discussions and consultation)

• ... And other organizations that didn't want us to report on this publicly

Note 7 Mar 2023: I just started this page, it is far from complete

Organizations and 'do' initiatives

: Including (new) EA-aligned marketing groups

EA-related giving initiatives

Non or semi-EA initiatives

EA or 'effective giving' orgs working with foundations and wealthy donors

Research and information-gathering initiatives

We are an academic collective and research non-profit, dedicated to providing public communication campaigns with cutting-edge research and rigorous tools for message development.

"Crowdsourcing" ... Recent research suggests that regular people can often be far more effective than experts at predicting which messages will best resonate with others in their community.

-->

• Efficient message search. We design research pipelines that allow campaigns to explore the large space of potential messages more efficiently, and to quickly zero-in on the most impactful messaging strategies. Our methodology is based on a combination of large-scale adaptive online survey RCTs, Bayesian machine learning and surrogate metrics.

(next subsection: )

Other marketing/implementation resources

• : Design and implementation of marketing; EA aligned and savvy.

  • See

• JS Winchell offers advice on how to do this right and leverage certain grants and do good marketing.

• JS Winchell has started an agency called ""; they are working to implement and measure the impact of EA marketing for the highest-value clients.

• provides free tech support and development to organizations in the (EA) community.

Contexts to test outreach messages

Contexts allowing individual randomization & tracking of medium-term outcomes

1. GWWC web site at point of email signup

2. Email lists

   • immediate: subject headers w/ 'open rates' as dependent variable

   • medium-term: all outcomes tied to email

Contexts for 'Immediate outcomes' (clicks etc)

Facebook; But the targeting algorithm may frustrate randomization. (see .) Can it be switched off?

Contexts allowing

See

This is helpful if the important outcomes can be tracked by ZIP code/post code/address.

• Online display advertising

• Google search

• YouTube

• LinkedIn

• Facebook (presumably)

Testing Rich Content

How to test rich content?

We can use some of the same strategies as above to test "rich content", i.e., short or even long talks, book chapters, podcasts, and so forth.

Targeting

Question: If our aim is to change the culture of giving in general, what kind of people should we be targeting?

1. Influencers (People with lots of social influence)

2. Low-hanging fruit (i.e., people who are naturally predisposed towards effective giving, pledging, & EA)

Ideas/methods for targeting: platforms and audiences

For a trial to yield insight, we need to be able to track and measure meaningful outcomes, and connect these to the particular 'arm' of the trial the person saw ... (if they saw any arm at all)

Notes from conversations (need explanation)

In this section we discuss how to see the results of your promotions and trials, and how to access data sets of these results that you can analyze.

• Facebook ads interface and Pivot tables

• Google A/B, optimize interface and Google analytics interface

Cost of ads: benchmarks

Reinstein, FB ads tied to fundraisers.

My costs have been:\

• about $0.01 per impression

• about $0.50 - $1.20 per click

Targeting at Universities ... Facebook's estimates

The estimated cost per impression (?‘reach’) and per click varies with the targeted audience. In general, narrower is estimated to be more costly. I think this is about ‘a larger audience allows FB to serve the ads to a larger number of people who tend to be click-happy’ Some data points:

• For Oxford, ‘In College’, living in the UK: They claim we will get 4-18 clicks per day for $50 per day over 2 days (29 Mar 2022 check on FB ads manager)

• If I put in Birmingham instead I get a fairly similar figure.

• If I remove the only-one-university narrowing, it gets cheaper. They claim I’ll be able to get 86-250 clicks per day for the same cost …

Meta allies

Research advancement manager: Michael Zoorob

EA groups (employees) within Meta

If you run a lot of ads FB will assign you external consultant helpers. They are somewhat helpful, but they don't seem to know everything.

FB tips for charities/fundraisers

28 Nov 2022, Zoroob:

Meta has just released a recorded series of videos (Meta’s Nonprofit Advertising Education Series) to help non-profit organizations meet their year-end fundraising goals. Some of these materials may also be helpful for researchers using Meta ads (e.g., materials on designing effective ad creatives), so I am passing the info along. Blurb below.

The three-part series of virtual webinars provides nonprofits with advertising training and best practices around how to use Meta technologies to further their missions:

Advertising Basics: Get started with Meta advertising with our Nonprofit Advertising page today.

The session also features On-Facebook Donation Ads that enable donation transactions within the Facebook app.

Creative Best Practices: Learn what great nonprofit creative can look like with best practices from Meta.

Consider saving our Creative Considerations guide to learn more about the five key creative considerations that apply to cause-driven campaigns.

Measurement: Introduce yourself to measurement best practices on Facebook and Instagram! Afterwards, explore split testing, lift measurement, and the experiments tool, on our Advertising Measurement hub.

Visit our website to view resources to help your organization drive more positive change, and do more good.

These and other videos for non-profit advertisers can be found on the "On-Demand Video Library."

Other links and issues

Facebook ads interface (collecting data)

Facebook split-testing issues

Facebook Fundraiser API Seems particularly useful but access is limited; they hope to make it more generally available some thing like mid- spring 2023.

Google ads grants

JS Winchell presentation:

Tomasik (2014) post

General grants -- see "grants" section

Opinion: Digital marketing is under-utilized in EA: JS Winchell Post

Videos - Facebook

Jamie Harris Summary

Youtube seminars

Email from JS:

The YouTube team holds quarterly workshops to explain how best to build and use your organic (not paid) YouTube channel. Based on previous discussions it sounds like this is something that might be of interest to your orgs.

Note: This is aimed at beauty and fashion brands but I'd imagine 80% of it would apply to GWWC/80k/1ftw

Agenda for the workshop:

• Explain why YouTube is crucial for your brand identity

• How to claim your narrative on this platform

• Reach and engage new and existing audiences through content

• A review of channel best practices

• Enhance your channel's search and discovery potential

• Develop an always on strategy

Register your interest for future workshops here.

Using Meta ads manager reporting suite

Extracting simple results

1. Go to "the reporting suite in Meta ads manager"

2. Specify some filters:

This gets us the screen below

3. Specify the date range.

5. Export simple results for Campaigns

Click 'Reports' ... upper right.

We can 'create a custom report', which saves this for later tweaking, or merely 'export table data'. I will do the latter for now:

Note: I chose CSV and do not include summary rows, to avoid confusion later.

Now I import this data into R (I usually use code but let's do it the interactive way for illustration)...

It seems that the option 'include summary row' was probably not wanted here, and that row with blank 'campaign name' could cause confusion.

It seems to have removed the "bid strategy" column, and added 'reporting starts' and ...'ends' from the filter. Otherwise, everything else seems the same as in the ad manager view, although some labels have changed.

Campaigns, ad sets, ads

We see three tabs

1. Campaigns

2. Ad sets for 1 campaign

3. Ads for 1 campaign

Campaigns

Here we have 7 campaigns, each with separate budgets, and start and end dates (although these mainly overlap).

It looks like some campaigns were set up for direct comparison or "A/B" perhaps, with the exact same budgets and end dates, and similar names:\

Ad sets

Here, there are 52 total 'ad sets' across all campaigns.

I'm going to export this as a csv too, in case it's useful.

Ads

There are also 52 "ads"; it seems in this case, one per ad set:

The information in the 'ads' table seems the same as in the 'ad sets table' ... other than a link to preview the ad content itself (which I don't seem to have access to atm).

Below, we give one example from a relevant context, illustrating (with screenshots) what choices you might make, what it would look like, and how to implement it.

See also: Facebook split-testing issues and #videos-facebook

Getting started

"Meta Business Suite"(https://business.facebook.com/) is the starting point of your ad campaign. If you have a Facebook Business account, you should have a "Meta Business Suite":

Next, click on "Ads manager" (See the megaphone on the left).

Budget optimization

When creating a new "Traffic campaign" ('cold traffic campaign' referenced HERE) there are a lot of options to help you optimize your delivery while minimizing your expenses.

You need to opt-in to these tools by ticking "create A/B test" and "Budget Optimization" on the first page of your "ad campaign manager." Since there is no downside (we would like to learn which ad design works best), we decide to opt-in to each of these.

Budget optimization is closely related to the choice of the target group. In general, the larger the target group, the cheaper it becomes to reach a certain amount of "link clicks".

Targeting the ad

Suppose we wish to create a targeted ad for a particular Facebook audience. For example, we might wish to put an ad...

• in the 'feed' of US Americans who are interested in charity or volunteering or philosophy

• giving them a link to a page encouraging them to learn about EA

We can use the "schedule and duration" function not only to automate the timing of our campaign, but also to estimate its cost. For example, we assume that we need 800 participants to click-through to start the 20 fundraisers (i.e., a rate of 2.5%).

Below, we see that FB estimates 172-497 link clicks per day for 10 Euros per day for (a different_ case.

You can specify

1. Demographics

2. Interests

3. Behaviors

"Include" seems to be the default when specifying these ... it 'expands the audience'. You can click 'narrow further' to constrain the audience.

How should we (EA, effective giving) target ads?

We have some evidence that narrower targeting helps. An obvious candidate is

Traffic choice

The next big choice is 'where do you want to drive traffic?'. You'll enter more details about the destination later.

Since we want people to click our web app, we chose "website".

Version testing

We may have several versions of the ad we want to try out, and we want Facebook to iterate towards the one that is more successful using their algorithm. Ideally, we would like to learn as much as we can about 'which ads perform better on which audiences'.

We can set up Facebook's ("meta") algorithm to dynamically optimize 'over which will get the most clicks.'

Where do we actually specify, enter, and style our ad content?

Finally, we have to decide which delivery we want to optimize.

We may want the ad that gets the most "conversions traffic to our page". Therefore, we choose the option "link clicks".

However, we might instead want FB to optimize the ad presentation in terms of which ad not just leads to the most 'clickthroughs' but leads to the most "conversions" or some other action taken on our page To do that we need to set up a "meta pixel". See #optimizing-and-pixels

Cost and cost controls

DR: In my past experience, you ended up paying Facebook based on the number of "clicks" you got not simply on how long your ad was up. But it's probably a combination of these, and there are probably different pricing plans. You can tell Facebook to put a limit on either of these do not go "over budget". Facebook will aim to spend your entire budget and get the most link clicks using the lowest cost bid strategy.

Currently EUR 315 is the max for new users ... but for our present pilot we may want less than this (check: how much do we expect to pay for 800 clicks, let's split this up into ... first 100 clicks, next 300 clicks,.. to see if its going OK )

Designing your ad

Finally, you enter the third and last page of the ad creation process. Here you have to verify your ID and Facebook page and choose the actual design of your ad versions. ["of which the most important one is whether you want to have a video or single image." (?) ]

The last step before publication is to specify the destination for your campaign.

We chose a website and simply copy the URL into the mask to make sure the ad is linking people to the right destination.

Payment (and monitoring)

Optimizing and pixels

Setting up the pixel

The pixel includes content from Facebook that needs to be integrated into your website/page of interest. (To do: link instructions for this).

Adding pixel 'events' to your web page

One simple way of doing this: "Events setup tool"

Once you are in the ads manager for an ad, go to the 'Events Manager':

"Add events", choose "from the pixel"

"Events setup tool"

Put the URL for your site in and 'Open website'

As seen below, this opens our page, and show what things have already been associated with a Pixel. Here the "create fundraiser" button on this page has been associated with a button on this page with the "Initiate Checkout". (We use default names Facebook is familiar with, even though there is no 'checkout' in this case).

("Facebook Pixel Helper" extension in Chromium might be helping here, but I'm not sure how).

For example, I could click 'who are we' on a page and associate it with 'view content'

I could 'add a value' to this, if it makes sense.

After this 'finish setup' ... it gives you the chance to see what you have asked it to do and confirm or cancel it.

Using the pixel events for Facebook ad optimization

Once you have nice pixels set up, you can use this in helping Facebook decide which versions of ads to serve, which audiences to serve them to, etc. You set up your ad, define an objective etc...

Here we're choosing 'initiate checkout', which we defined as clicking on a 'create fundraiser' button on the first page of our site (early in the funnel)

The warning below might not matter as we

I assume that the same 'conversions' target defined above is used in optimizing the 'dynamic creative' if you turn that on.

Add section: How to set up GA

Tracking the progress of a trial in GA

Some key 'flows and tips'

• **'**Home'

• 'behavior', 'site content', 'all pages'

  • remember to set date range!

• Acquisition, all traffic, channels: here 'social' (probably) tells you who came from Facebook etc

  • Acquisition, all traffic, Source/medium drills down into this

"Getting the data"

DR: I'm not sure how to get 'all the data', but I have been able to get data on, e.g.,

• a set of outcomes,

• over a set period of time, (a particular month and the same month in the previous year)

• broken down by another feature (by city)

Then search and select your desired ‘metrics’ (outcomes) of interest. “Users” and “sessions” seem pretty important, for example.

Next you can break this down by another group such as “city”. You can put in 'filters' too, if you like, but so far I don't see how to filter on outcomes, only on the dimensions or groups.

I don't know an easy way to tell it to “get all the rows on this at once.” but if you scroll to the bottom you can set it to show the maximum of 5000 rows.

Next, scroll up to the top and select export. I chose to Export it as an Excel spreadsheet., as this imports nicely into R and other statistical/data programs.

We were able to do this in two goes, but for larger datasets this would be really annoying. I imagine there is some better way of doing this., maybe a way of using an API interface for Google Analytics to just pull all of this down.

A partial workaround fix is to do a ‘filter’ to discard rows you don’t need… click ‘advanced’ at the top and…

Facebook's Ad Manager and Google Analytics often report results that seem to have discrepancies. Below, one particular case, and possible explanations.

What is going on 'in our latest trial'?

• Facebook: We have 50k+ unique impressions, and 1335 clicks

• Google Analytics records only 455 page views, 403 users

  • And only about 20 doing any sort of Engagement like scroll or click (if we read it correctly)

1. Where do the other 600 clicks end up? Ad blockers? Do they click the ad and shut down before the page comes up?

JS: main reasons [DR: slightly edited[

1. "Do they click the ad and shut down before page comes up?" Yup! Closing the page before the redirect fully loads. Facebook will be as generous as possible with their click reporting.

2. ... If a user clicks on the FB ad twice within 30 minutes, then Google Analytics would record that only as a single user and a single session.

3. If a user has JavaScript disabled or doesn’t accept cookies, then Google Analytics doesn’t track.

Leticia at Facebook: can be mistaken clicks, this is common.. need a pixel to fix this ..., can change it to 'landing page view'

2,. How is it possible that 455 people come to the page and only 20 (under 5%) of them actually even do anything on the page?

Why use survey/experiment/marketing platforms?

• To test content in more depth than an A/B trial permits

• Better control over 'who is participating' and how much attention they are paying

• Things more towards 'basic background research'

• Closer to a 'representative sample'

Some participant recruitment platforms

: Created specifically for (academic research). Our impression is that this is among the highest quality panels, although there is some controversy over this.

CloudResearch: CR approved Mturk

CloudResearch: Prime Panels

Positly: https://www.positly.com/ Qualtrics (panel) Lucid

Dyndata

Mapping the key non-obvious features of running and analyzing these A/B trials using the Google analytics/optimize system.

Reporting and considering this in the context of the GWWC

What is it? How to create a trial

Setup and requirements

Set 'variants' and give them weights (assign shares for each)

Page and audience targeting

Objectives: The things you are trying to measure and improve

"Activity": Trials created/started/changed/ended/archive

Viewing results and Google's analysis

Clicking on a particular 'experience' in the 'container'...

Optimize report: top

• I am not sure where we can learn 'how many users there were'.

• ("View full chart" can give you a day-by-day breakdown of the number of sessions.)

OR: Conversion rates section

The next section compares 'sessions' and 'conversions' by treatment, and does a Bayesian analysis. This seems the most useful part:

Relative conversion rates, analysis

Above, the 'Separate block' (SB) seems to be the best performing treatment. Google calculates a 2.69% conversion rate for this (here, presumably the rate of people checking 'any' of the follow-on boxes).

Considering the Analysis, Google Optimize "uses Bayesian inference to generate its reports... [and] chooses its priors to be quite uninformed." The exact priors are not specified (we should try to clarify this).

But if we take this seriously, we might say something like ...

if our initial priors gave each treatment an equal chance of having the highest conversion rate ('being best'), and assumed a [?beta] distributed conversion rate for each, centered at the overall mean conversion rate ...

then, ex-post, our posterior should be that the SB treatment has an 80% chance of being best, our 'Original' has a 17% chance of being the best ...

Google also gives confidence intervals for the conversion rates for each treatment, with boxplots and (95%) credible interval statistics:

The grey bar for the baseline is mirrored in all rows. The 95% CI for the 'improvement over the baseline' is given on the right. But this is a rather wide interval. More informatively, if we hover over the image, we are given more useful breakdowns:

Although this does not exactly tell us the 50% interval 'improvement over the baseline' (this would need a separate computation), we can approximately infer this.

But fortunately it is reported in data we can download; see below "Download (top right)".

From that data, we get:

Our 'posterior' probability thus infers (assuming symmetry, I think) that we should put (considering odds ratios, not percentage points)

• a 2.5% chance of SB having an 18% (or more lower rate of conversion than 'Original'

• a 22.5% chance on SB being between 18% worse and 4% better

• a 25% chance of being 4-20% better

• a 25% chance of being 20-36% better

• A 22.5% chance of being 36-76% better

• A 2.5% chance of being more than 76% better

We can also combine intervals, to make statements like ...

• a 50% chance of being 4-36% better

• a 50% chance of being 20-76% better

There is some repetition (can we 'mirror blocks'?)

Session balance

Above, even though the treatment has been assigned randomly (presumably a close-to-exact 1/3, 1/3, 1/3 split), the number of 'sessions' differs between the treatments ('variants').

Why? As far as I (DR) understand,

• while each individual user (at least if they are on the same machine and browsing with cookies allowed) is given the same treatment variant each time...

• the same users may 'end' a session (by leaving or being inactive for 30+ minutes), and return later, getting the same treatment but tallying another 'session'. This suggests that users in the "Separate Block" (SB) treatment are returning the most (but also see 'entrances' below).

Breakdown over time

The final section gives the day to day breakdown of the performance of each treatment, presumably, along with confidence intervals. This seems relevant for 'learning and improving while doing' but possibly less relevant for our overall comparison of the pages/treatments.

Download (top right)

The 'Analytics data' gives us sessions and conversions by day and by treatment.

(Where no session occurs in a day for a treatment, it is coded as blank).

Clicking on 'view in analytics'

... this gives some other information, mainly having to do with the user experience.

"Unique page views" represent "the number of sessions during which that page was viewed one or more times." ... Recall "sessions" are periods of continuous activity.

"Entrances" seem potentially very important. According to Google:

Sessions are incremented with the first hit of a session, whereas entrances are incremented with the first pageview hit of a session.

In the present context, this suggests that the 'Separate block' page is inspiring users to come back more often, and to spend more time on average.

Sessions vs. Users

As noted, essentially: 'Sessions' are the number of 'continuously active' periods of an individual user

Analytics measures both sessions and users in your account. Sessions represent the number of individual sessions initiated by all the users to your site. If a user is inactive on your site for 30 minutes or more, any future activity is attributed to a new session. Users that leave your site and return within 30 minutes are counted as part of the original session.

The initial session by a user during any given date range is considered to be an additional session and an additional user. Any future sessions from the same user during the selected time period are counted as additional sessions, but not as additional users.

Pivot tables

You may want to see or export crosstabs of one outcome, user feature, or design feature, by another. Sometimes you just want to see these quickly, but this might also be a way to extract the 'raw data' you wish to analyze elsewhere.

1. Start new pivot table

From within Ads Manager From 'ads reporting' (3 Aug 2022 updated interface)

Click "Create Report" --> Pivot table

2. As before, make sure you've selected the right date range, and (redo) any relevant filters

Here I add a filter for 'campaign name ' contains 'general'. Because I'm specifically trying to pull down some information on 'which video people saw' in this group (which needs a special setting to access... as noted below)

3. "Customize pivot table" – "Breakdowns" ... the things you want this to disaggregate across (sums and averages within groups)

• the 'campaigns', the 'ad names'

• timing, demographics

• Drill down to "Custom breakdowns", "Dynamic Creative Asset", to get it broken down by the text linked to the ads:

4. "Customize pivot table" – "Metrics"

Select the things you want reported, and deselect things that are not interesting or irrelevant to this case (like 'website purchases') or numbers that can be easily computed on your own

Normally, I'd suggest leaving out the redundant 'Cost per Result' but it's probably good to have as at least one sanity and data check.

Other stuff like 'video play time' could sometimes be very relevant, but I'll leave it out for now

5. (Optional) Conditional formatting

This could also be helpful if you are using the Ads Manager tools in situ, but obviously this has no value for downloading.

6. Save report for later use, share

If you think the report is useful in-situ, you can also share a link

7. Export the data

(or consider direct import into R using tools like the rfacebookstat package)

For each proposed/ongoing/past trial, let's try to report the following minimal details, with links (proposed template) If you don't have time and you have another clear presentation of most of this, please link or embed it.

"Concise reporting template"

General idea, main 'hypothesis' (if there is one)

Firstly, what is this promotion trying to do (e.g., 'encourage signups for giving pledges')?

But more importantly, what are you trying to learn here... What might you have a better understanding of after the trial than you did before the trial?

E.g.,

Specifically:

• Does the opportunity we offer to sign up for an 'accountability partner' increase or decrease the rate at which people DO XXX activity?

• Does it lead to greater overall XXXlinked donations per visitor over the next 1 year interval?

Generally:

• Does 'social accountability' help to encourage XXX activities and promises and the fulfillment of these? Does the 'fear of being held accountable' discourage people from making commitments?

(Optional: brief on background theory and previous evidence)

Point of contact (at organization running trial)

You can enter more than 1 person here, including an external organizer (like JS Winchell), but ideally, also someone inside the organization.

Add 'academic/research lead' here if there is one

Timing of trial (when will it/did it start and end, if known)

Digital location where project 'lives' (planning, material, data)

The present Gitbook/and a nested Github repo folder could be ideal. Please give a precise link so others could access it.

Environment/context for trial

(Is it on a web page, a google advertisement, a physical mailing, etc)

Participant universe and sample size

• Who will be targeted or who do you expect to be part of the trial?

• (Somewhat optional) How many people (or 'units') do you expect to be involved (median guess)?

• (Optional): How many do you expect will have a 'positive outcome' (e.g., a 'conversion')?

Key treatment(s)

Description, link exact language/content if possible

Treatment assignment procedure

• At what level is it varied? (individual visitors, postal codes, days of the week, etc)

• How are treatments assigned ('blocked randomization', 'adaptive/Thomson sampling', etc.)

  • If you are using a 'set Google, Facebook etc algorithm', just input the settings you used here, and/or link the (Google, FB, etc) explanation

• How many/what shares are assigned to each treatment?

Outcome data

• What measures (outcomes, other features) will be collected?

• When and how

• Where will the data be stored, who will have access

Optional/suggested additions

Planned analysis methods, preregistration link, IRB link, connection to other projects and promotions

Ex-post: Reporting results (brief)

Implementation and data collection

1. Did it go as planned? Any departures? (Timing, randomization, design changes, etc)

2. How much/what data was collected? How many observations?

3. Where is the data stored (also link/adjust the above), who has it, and under what conditions?

Basic results/outcomes

1. "Partners and stakeholders opinions": were they happy with the trial? Did they seem to think it was a success?

2. Simplest statement (e.g., "3% donated in the treatment versus 2.2% in the control, with an average amount raised of $4.3 in the treatment and $3.1 in the control')

3. Preliminary interpretation, with statistical test if possible (e.g., 'google Optimize states an 80% chance that the treatment outperformed the control', a Fisher's exact test yields a p=0.06 that a positive donations was more likely in the treatment than the control)

4. "Full analysis"

   1. Who/what when will it be done?

   2. Link to 'where' it will be done (both the 'follow up the pre-analysis plan, and the full write-up, if applicable)

   3. Possibly: Briefly characterize the overall confusions/state of analysis here (state the date last updated)

5. Feeding synthesis and meta-analysis

   1. Which generalizable questions does this inform?

   2. Is data sharable? Key comparable outcomes?

   3. What other work/trials does this relate to?

   4. State of meta-analysis

Estimating Ad Effectiveness using Geo Experiments in a Time-Based Regression Framework Jouni Kerman, Peng Wang, and Jon Vaver Google, Inc. March 2017

Abstract .... While effective, this geo-based regression (GBR) approach is less applicable, or not applicable at all, for situations in which few geographic units are available for testing (e.g. smaller countries, or subregions of larger countries) These situations also include the so- called matched market tests, which may compare the behavior of users in a single control region with the behavior of users in a single test region. To fill this gap, we have developed an analogous time-based regression (TBR) approach for analyzing geo experiments. This methodology predicts the time series of the counterfactual market response, allowing for direct estimation of the cumulative causal effect at the end of the experiment. In this paper we describe this model and evaluate its performance using simulation.

• DR hypothesis note-taking version

• Related software package

See Geographic segmentation/blocked randomization for a mainly theoretical discussion of this

Facebook split-testing issues for how to do split testing on Facebook, and the limits to traditional design given their setup

Naturalness of setting versus 'pure' treatments

Academics usually try to make each treatment differ in precisely one dimension, these treatments are meant to represent the underlying model or construct as purely as possible. This can lead to setups that appear strange or artificial, which itself might bring responses it will not be representative or generalizable.

When we consider an experiment providing 'real impact information' to potential donors, we might be encouraged to use the exact write-up from Givewell's page, for naturalness. However, this may not present the "lives per dollar" information in exactly the same way between two charities of interest, and the particular write-up may suggest certain "anchors" (e.g., whole numbers that people may want to contribute). Thus if we use the exact GW language we may not be 100% confident that the provision of the impact of information is driving any difference. We might be tempted to change it; but at a possible cost of naturalness and direct applicability.

There are very often tradeoffs of this sort.

Awareness of testing can affect results

In the present context, we have posted about our work, in general terms, on a public forum (EA forum post). Thus the idea that ‘people are running experiments to promote effective giving and EA ideas’ is not a well-kept secret. If participants in our experiments and trials are aware of this it may affect their choices and responses to treatments. This general set of problem is referred to in various ways, referring to different aspects of this; see 'experimenter demand', 'desirability bias', 'arbitrary coherence/coherent arbitrariness', observer bias (?), etc.

Mitigating this, in our context, most of our experiments will be conducted in subtle ways (e.g., small but meaningful variations in EA-aligned home pages), and individuals will only see one of these (with variation by geography or by IP-linked cookies). Furthermore, we will conduct most of our experiments targeting non-EA-aligned audiences unlikely to read posts like this one. (People reading the EA forum post are probably ‘already converted’.)

Incentives and 'more meaningful responses'?

Other issues to consider

(To be fleshed out in more detail)

• Universe (population) of interest, representativeness

• Design study to measure 'cheap' behavior like 'clicks' (easier to observe, quicker feedback) versus meaningful and long-run behavior (like donations and pledges)

  • attribution issues

  • attrition issues (also see the quantitative sections)

• Choice of impact measure/metric (also see the quantitative sections)

Measurement needs are varied and come with a variety of limitations, e.g., data avail-ability, ad targeting restrictions, wide-ranging measurement objectives, budget availability,time constraints, etc

Kerman et al, 2017

Why 'Geo experiments'

In many contexts, the route to a meaningful outcome (e.g., GWWC pledge) is a long one. Attribution is difficult. An individual may have been first influenced by (1) YouTube ad while seeing a video on her AppleTV, and then (2) by a friend's post on Facebook, and then finally moved to act (3) after having a conversation at a bar and (4) visiting the GWWC web site on her telephone.

The same individual may not (or may) be trackable through 'cookies' and 'pixels' but this is already very limited and imprecise, and is being made harder by new legislation.

"Geographic targeting" of individual treatments/trials/initiatives/ads may help better track, attribute, and yield inference about 'what works'. E.g., we might do a 'lift test':

1. select a balanced random set of US Zip codes for a particular repeated YouTube ad promoting GWWC, the "Treated group"

2. compare the rate of GWWC visits, email sign-ups, pledges, and donations in the next 6 months from these zip codes relative to all other zip codes. (Possibly throwing out or finding a way to draw additional inference from zip codes adjacent to the treated group)..

We could also do multi-armed tests (of several types of ad or other treatment, with a similar setup as above)

There are a few well-known and researched approaches: From Kerman et al, 2017 (emphasis added)

Geo experiments (Vaver and Koehler, 2011, 2012) meet a large range of measurement needs. They use non-overlapping geographic regions, or simply “geos,” that are randomly, or systematically, assigned to a control or treatment condition. Each region realizes its assigned treatment condition through the use of geo-targeted advertising. These experiments can be used to analyze the impact of advertising on any metric that can be collected at the geo level. Geo experiments are also privacy-friendly since the outcomes of interest are aggregated across each geographic region in the experiment. No individual user-level information is required for the “pure” geo experiments, although hybrid geo + user experiments have been developed as well (Ye et al., 2016). Matched market tests (see e.g., Gordon et al., 2016) are another specific form of geo experiments. They are widely used by marketing service providers to measure the impact of online advertising on offline sales. In these tests, geos are carefully selected and paired. This matching process is used instead of a randomized assignment of geos to treatment and control. Although these tests do not offer the protection of a randomization experiment against hidden biases, they are convenient and relatively inexpensive, since the testing typically uses a small subset of available geos. These tests often use time series data at the store level. Another matching step at the store level is used to generate a lift estimate and confidence interval.

Where and how can we geographically block treatments?

What if we can only apply the treatment to one, or a few, of many groups?

We still mahy be able to make valuable inferences, under specified conditions, through 'difference in difference', 'event study', and 'Time based' approaches. We consider this in the next section: Difference in difference/'Time-based methods'

Sections

"Qualitative" design issues: How to design the 'content' of experiments and surveys to have internal validity and external generalizability

Real-world assignment & inference: How to set up trials to have comparable groups

Adaptive design/sampling, reinforcement learning: Adjusting the treatments and design as you learn, to 'get to the highest value in the end'

Analysis: Statistical approaches: How to make inferences from the data after you have it (and plan this in advance)

What are our estimation goals?

Statistical power versus optimized learning

Fixed vs adaptive designs

See adaptive design notes

Resources

Rethink Priorities notes (some are works in progress)...

The https://declaredesign.org/ framework and R package seems very helpful. I (David Reinstein) am learning and trying to adapt it.

Dillon's 'Hemlock'

Reinstein 'research tools and data' airtable list

Facebook trials: "divergent delivery" --> limited inference

Researchers are interested in running trials using Facebook ads. However, inference can be difficult. Facebook doesn't give you full control of who sees what version of an advertisement.

1. With A/B split testing etc: They have their own algorithm, which presumably uses something like Thomson sampling to optimize for an outcome (clicks, or a targeted action on the linked site with a 'pixel'). Statistical inference is challenging with adaptive designs and reinforcement learning mechanisms. As the procedure is not transparent, it is even more difficult to make statistical inferences about how one treatment performed relative to another.

2. Segmentation and composition of population: Facebook's 'PageRank' algorithm determines who sees an ad. I don't think you can turn this off.

   1. We haven't found a way to be able to set it to "show all versions of an ad to comparable populations"

   2. (And even if you could, it would be difficult for you to specifically describe "which population" your results pertain to.)

Divergent delivery and "the A/B test deception"

Example: "Hours spent promoting" vs "number of fellows"

Consider a study where

• EA groups are asked to voluntarily participate (with no direct compensation)

• to report the 'time spent on each recruiting activity',

• and to ask their fellows/members 'how did you hear about our group?'

Suppose this finds

'per hour spent by the organizers, far fewer people report "tabling" as the source, relative to 'a direct email'.

Should we interpret this as

'direct emails are a more efficient use of time than tabling, thus groups should spend less time doing tabling and more time sending emails?'

Maybe, but we should be careful; there are other explanations and interpretations we should delve into. Some of these could be partially addressed through survey design, others through careful analysis. Other 'causality' issues may require an experiment/trial/test to get at.

Statistical inference: chance and selection/selectivity

Random variation: With a small sample of groups, these numbers may be particularly high or low (for tabling, for emails, etc) by chance; the averages for a 'typical group' may turn out to be very different.

• This is the standard issue of statistical inference about a population from a sample.

• The issue of 'misrepresenting the population' tends to be worse with smaller samples (here small number of groups, and small numbers of observed outcomes in each group; e.g., only a few fellows)

• However, 'as Bayesians know' you can still draw valuable decision-relevant inferences from small samples. IMO (Reinstein) the "problem of small samples" tends to be overstated because we mainly learn about statistics designed for a particular scientific frequentist approach.

Selection/selectivity: The groups that 'opted in' to be part of this survey may not be a 'random draw' from the population of relevant groups. It may represent more careful or more enthusiastic groups, perhaps groups that are particularly analytical and not so good socially, etc. If some of the 'fellows' within the groups don't complete the survey, this could add another 'selection bias'.

'Marketing causality' issues

• Attribution with multiple sources: “How did you hear about this program?” This could be interpreted in several ways, probably “how did you first hear”. But in marketing sometimes people hear about something multiple times, and it’s hard to know which of these are pivotal in getting them to take action. (We could probably do something to make this question a bit more informative.)

• “Lift”: some people might have signed up anyways even without the activities identify as ‘how they heard about it’. Other people may have been harder to reach, and for the latter (e.g.) tabling ‘Spoke to us while we were tabling’ may be pivotal.

Decisionmaking implications

• Diminishing returns/hard limits on some activities … e.g., there may be only so many professors (or students) to email. After a few hours of this

Impact of treatment on 'rare event' incidence

Notes from slack:

I’m finding some issues like this in analyzing rare events … not quite that rare, but still a few per thousand or a few per hundred.

I’m taking 2 statistical approaches to the analysis (discussion, code, and data in links):

1. Bayesian binomial-beta (a pretty standard setup I’m probably making overcomplicated)

2. Randomization inference (simulation) … for a sort of equivalence testing here

I think either of these could be ‘flipped around’ to be used for power calculation or ‘the Bayesian equivalent of power calculation’

My colleague Jamie Elsey has some expertise with the latter; we’re putting together our discussion HERE, although it’s mainly frequentist and not Bayesian ATM.

Open and robust science: Preregistration and Preanalysis plans

There are reasons 'some pre-registration' or at least 'declaring your intentions in advance' is worth doing even if you aren't aiming at scientific publication

Which statistical tests/methods

https://gitlab.com/dsbowen/conditional-inference/-/blob/master/examples/bayes_primer.ipynb

From Sample to Population: Multilevel Regression and Poststratification (MRP) and Survey Weighting

Overview: conversation with Dillon Bowen

Overview: conversation with DB

Dillon Bowen: End of 3rd year of decision processes in Wharton PHd.

Here is a stats package for estimating effect sizes in multi-armed experiments. https://dsbowen.gitlab.io/conditional-inference/

Adaptive experimentation software: Hemlock

I just made a getting started video: Welcome to Hemlock - YouTube

Adaptive experimentation (discussion)

...running experiments with many arms and winnowing out the 'best ones' to learn the most/best.

• See: adaptive design, adaptive sampling, dynamic design, reinforcement learning, exploration sampling, Thompson's sampling, Bayesian adaptive inference, multifactor experiment

Treatment space

Discrete vs continuous: switches vs. knobs

In our cases of the ‘options are discrete’, many knobs to turn, although some are discrete. There is a different version of this for discrete vs continuous

If we can order the different treatments (arms/knobs) as 'dimensions' we can infer more... Can do better thinking of them as a ‘multifactor experiment’ rather than 2 unrelated … several separate dimensions

"Model running in the background" trying to figure out ‘things about the effectiveness of the interventions you might use’

'Explore only' or 'explore & exploit' at the same time

“Ex-post regret versus cumulative regret” … latter suggests Thompson sampling (Does Thompson's sampling take into account the length of the future period?)

Learning and inference

Ex-post … Use machine learning to consider which characteristics matter and how much they matter … although he doesn’t know of papers that have looked at this, but assumes there are adaptive designs that incorporate this.

Statistical inference can be challenging with adaptive designs, but this is a ripe area of research

Dillon: has a paper on traditional statistical inference after an adaptive design.

Goals 'what kinds of inference':

1. The arm you using relative to (? the average arm?)

2. Which factors matter/joint distribution ….. Bayesian models

Resources

Todo: Integrate further easy tools and guides, including those from Jamie Elsey

"Even a few observations can be informative"

Drawing from Lakens' excellent resource:

You are considering a new and an old message.

Suppose you are a ‘believer’ … your prior (light grey up to) is that ‘this new message nearly always performs better than the control treatment’

Suppose you observe only 20 cases and the treatment performs better only half the time. You move to the top black line posterior. You put very little probability on the new message performing much better than the control.

Now suppose you have the ‘Baby prior’, and think all of the following ten things are equally likely

• less than 10% of people rate the new message better than the control

• 10-20% of people rate the new message better than the control

• …

• … 50-60% of people rate the new message better than the control

• ...

• 90-100% of people rate the new message better than the control

You run tests on 20 people, and you get 15 people preferring the new message.

Now you update substantially. From some calculations (starting from Lakens' code, pbeta(0.65, aposterior, bposterior)) you put about an 80% posterior probability that the new message is preferred by at least 65% of the population. (And only about 1.5% probability on the control being better)

So if I really ‘am as uncertain as described in the example above’ about which of two messages are better (and by how much)...

... then even 20 randomly-selected people assessing both messages can be very informative. How often does this ‘strong information gain’ happen? Well, under the "baby prior", you would get information at least this informative in one direction or the other about half the time.

Strategic considerations

Previous sections considered... 'How to get more people to care about '. 'How to get the "Einsteins" of the next generation interested in this.' And 'how do we introduce this to people?'

But, an equally-important concern may be... WHOM do we target? How do we do market profiling? Not just 'what do we present', but 'who do we present it to'

In this section, we cover the limited work that has been done on this, and the scope to do more.

Scoping and considering the value of doing this

Leander Rankwiler's recently (17 Feb 2023) did a scoping exercise for this. See . This work focuses on "the rationale, literature research, and data collection", and comes to relatively negative conclusions ("it's much less valuable to pursue than previously assumed"). This particularly reflects concerns that doing, publicly reporting, and acting on this research to 'target promising groups' may do some harm (see fold).

He also sees many sources of (statistical) bias in any feasible analysis.

In the sections below, we present and link recent and ongoing direct work that may also be relevant and informative.

RP: "How many people have heard of EA" survey

Asks respondents to tick terms and people that they are familiar with (EA/non, real/rare/fake). If they have heard of EA, we follow-up with open-ended questions to detect actual understanding. We also ask about socio-demography and politics. Administered to a ‘national sample’.

(We will follow up with attitude surveys among those who have heard of EA.) We use Bayesian models to generate the posterior distributions of

• share who know/understand EA within different groups,

• weighted to be nationally representative (of each group).

\

Wild Animal Welfare/Suffering attitudes

Various survey projects ongoing

EA Attitudes and Longtermist Attitudes

• Developing measures of attitudes towards EA/Longtermism

• Conducting large national surveys looking at predictors of these attitudes (including differences across groups)

• Standard ‘message testing’ (what arguments/framings work best for outreach (including differences across groups)

__

See

Rethink Priorities #ea-attitudes-and-longtermist-attitudes

RP has a remit and some funding to pursue this.

23 May 2022 update: After a long period with little relevant data, some work is underway from other groups: See sections below

Google Analytics (and other tools) collects or predicts the demographics and market profile of traffic on sites like givingwhatwecan.org.

This gives us a sense of the

• 'existing demographics' of those committed and/or interested ... in ways not picked up in (e.g.) the EA survey

• who is 'interested but not committed' ... possibly low-hanging fruit

Caviola et al

Farmed/overall

Faunalytics

This is possibly the best meta-resource as well as a source of original research

Sentience institute

Our Animals, Food and Technology (AFT) survey tracks attitudes towards animal farming and animal product alternatives in the US. In 2020, as in the 2017 and 2019 iterations, we found significant opposition to various aspects of the animal farming industry, with a majority of people reporting discomfort with the industry, and strong support for a range of quite radical policy changes, such as banning slaughterhouses. The trend in attitudes between 2017 and 2020 is relatively stable, though slightly negative (not statistically significant). Notably, the number of people who consider animal farming to be one of the most important social issues fell from 2017 to 2019, and remained at this lower level in 2020.