What is the "EA Market Testing Team"?

We are a . This project is organized by , who maintains this wiki/Gitbook and other resources.

We aim to promote awareness and understanding of the core ideas of Effective Altruism (EA), and "to make giving effectively and significantly a cultural norm." We consider marketing campaigns, charitable appeals, events, and public communication, working both with our and in independent surveys and trials. We want to improve the design and messaging of organizations like and to improve their outreach methods and maximize their impact. Measuring and testing 'what works and when': While helping these organizations do marketing and communication we are also testing and analyzing this rigorously. We help run and track careful data collection and rigorous controlled trials, as well as helping to organize the reporting of less rigorous trials. We robustly analyze the results to better understand which approaches tend to have a more positive impact.

• How to and test marketing campaigns?

• to promote EA?

• What ?

We strive to be transparent. We want to report and share our data, procedures, code, and evidence without overselling the results.

We believe this is the first organized collaboration of its kind. We aim to...

• Coalesce our understanding and evidence on barriers and facilitators of effective altruism, effective giving, and effective action

• Run a broad set of high-powered trials (large samples, high-stake real-world contexts, substantial differences between conditions)

  • ... to gather evidence on what works best to promote meaningful actions in specific

What is our mission? Updating...

As EAMT has progressed, we have encouraged others to do work and pursue initiatives in the 'space' of studying EA messaging, and marketing EA and effective giving. We hope that the resources we have provided, and the connections we have made have contributed to this. As the space changes, the EAMT mission, scope, and activities are adjusting as well.

We are moving towards a heightened focus on

1. Advising, proposing, and helping to design and coordinate experiments, trials, and initiatives.

2. of the results, rigorous statistical analysis

3. Synthesizing, sharing, and communicating this knowledge and skills base

This work provides substantial , whose benefit is shared among the partner organizations and the EA community.

Other relevant/new organizations and initiatives

• (including User Friendly, Good Impressions, and Altruistic Agency)

• Giving What We Can '' (a project we have encouraged and advised)

We believe the EA Market Testing Team is the first organized collaboration of its kind.

Goals and FAQ

(below in detail, scroll outside margin to skip past it)

What have we accomplished?

For an overview of our progress and ongoing work, see the we are building. (Below in detail, scroll outside margin to skip past it)

Note that we cannot publicly share details of ongoing and upcoming trials. We aim to share the results when it is possible. We aim to integrate shareable aspects of

"Testimonials"

How to get involved?

If you are interested in getting involved with our project or have feedback for us, contact David Reinstein at daaronr AT gmail.com.

Next, check out the .

(For an explanation of this Gitbook's structure, content, and aims.

1. This quote comes from the 2022 .

2. However, we are also careful to be efficient, recognizing the tradeoffs between rigorous experimental design and practical marketing.

3. Including , guides and

Opinion: Digital marketing is under-utilized in EA: JS Winchell Post

Videos - Facebook

Youtube seminars

Email from JS:

The YouTube team holds quarterly workshops to explain how best to build and use your organic (not paid) YouTube channel. Based on previous discussions it sounds like this is something that might be of interest to your orgs.

Note: This is aimed at beauty and fashion brands but I'd imagine 80% of it would apply to GWWC/80k/1ftw

Agenda for the workshop:

• Explain why YouTube is crucial for your brand identity

• How to claim your narrative on this platform

• Reach and engage new and existing audiences through content

Register your interest for future workshops .

Doing and funding ads

Video ads/Best-practice guidelines

Targeted ad on FB, with variations: setup (next subsection: Collecting outcome data)

Other marketing/implementation resources

• : Design and implementation of marketing; EA aligned and savvy.

  • See

• JS Winchell offers advice on how to do this right and leverage certain grants and do good marketing.

Below, we give one example from a relevant context, illustrating (with screenshots) what choices you might make, what it would look like, and how to implement it.

See also: Facebook split-testing issues and Videos - Facebook

Getting started

"Meta Business Suite"() is the starting point of your ad campaign. If you have a Facebook Business account, you should have a "Meta Business Suite":

Next, click on "Ads manager" (See the megaphone on the left).

The next step is to select "Create a campaign" and choose an "objective"... the interface gives you some idea of what these aim for:

Budget optimization

When creating a new "Traffic campaign" ('cold traffic campaign' referenced ) there are a lot of options to help you optimize your delivery while minimizing your expenses.

You need to opt-in to these tools by ticking "create A/B test" and "Budget Optimization" on the first page of your "ad campaign manager." Since there is no downside (we would like to learn which ad design works best), we decide to opt-in to each of these.

Budget optimization is closely related to the choice of the target group. In general, the larger the target group, the cheaper it becomes to reach a certain amount of "link clicks".

Targeting the ad

Suppose we wish to create a targeted ad for a particular Facebook audience. For example, we might wish to put an ad...

• in the 'feed' of US Americans who are interested in charity or volunteering or philosophy

• giving them a link to a page encouraging them to learn about EA

We can use the "schedule and duration" function not only to automate the timing of our campaign, but also to estimate its cost. For example, we assume that we need 800 participants to click-through to start the 20 fundraisers (i.e., a rate of 2.5%).

Below, we see that FB estimates 172-497 link clicks per day for 10 Euros per day for (a different_ case.

You can specify

1. Demographics

2. Interests

3. Behaviors

"Include" seems to be the default when specifying these ... it 'expands the audience'. You can click 'narrow further' to constrain the audience.

During this process, you can see a concise statement of your choices, and the estimated audience size further up on the page:

How should we (EA, effective giving) target ads?

We have some evidence that narrower targeting helps. An obvious candidate is

Traffic choice

The next big choice is 'where do you want to drive traffic?'. You'll enter more details about the destination later.

Since we want people to click our web app, we chose "website".

Version testing

We may have several versions of the ad we want to try out, and we want Facebook to iterate towards the one that is more successful using their algorithm. Ideally, we would like to learn as much as we can about 'which ads perform better on which audiences'.

We can set up Facebook's ("meta") algorithm to dynamically optimize 'over which will get the most clicks.'

Where do we actually specify, enter, and style our ad content?

Finally, we have to decide which delivery we want to optimize.

We may want the ad that gets the most "conversions traffic to our page". Therefore, we choose the option "link clicks".

However, we might instead want FB to optimize the ad presentation in terms of which ad not just leads to the most 'clickthroughs' but leads to the most "conversions" or some other action taken on our page To do that we need to set up a "meta pixel". See

Cost and cost controls

DR: In my past experience, you ended up paying Facebook based on the number of "clicks" you got not simply on how long your ad was up. But it's probably a combination of these, and there are probably different pricing plans. You can tell Facebook to put a limit on either of these do not go "over budget". Facebook will aim to spend your entire budget and get the most link clicks using the lowest cost bid strategy.

Currently EUR 315 is the max for new users ... but for our present pilot we may want less than this (check: how much do we expect to pay for 800 clicks, let's split this up into ... first 100 clicks, next 300 clicks,.. to see if its going OK )

Designing your ad

Finally, you enter the third and last page of the ad creation process. Here you have to verify your ID and Facebook page and choose the actual design of your ad versions. ["of which the most important one is whether you want to have a video or single image." (?) ]

The last step before publication is to specify the destination for your campaign.

We chose a website and simply copy the URL into the mask to make sure the ad is linking people to the right destination.

Payment (and monitoring)

Optimizing and pixels

Setting up the pixel

The pixel includes content from Facebook that needs to be integrated into your website/page of interest. (To do: link instructions for this).

Adding pixel 'events' to your web page

One simple way of doing this: "Events setup tool"

Once you are in the ads manager for an ad, go to the 'Events Manager':

"Add events", choose "from the pixel"

"Events setup tool"

Put the URL for your site in and 'Open website'

As seen below, this opens our page, and show what things have already been associated with a Pixel. Here the "create fundraiser" button on this page has been associated with a button on this page with the "Initiate Checkout". (We use default names Facebook is familiar with, even though there is no 'checkout' in this case).

("Facebook Pixel Helper" extension in Chromium might be helping here, but I'm not sure how).

"Track new button" lets you see what click options you could associate with a pixel.This highlights clickable things you can do this with. ('Create fundraiser' is not highlighted, probably because it's already been assigned).

For example, I could click 'who are we' on a page and associate it with 'view content'

I could 'add a value' to this, if it makes sense.

After this 'finish setup' ... it gives you the chance to see what you have asked it to do and confirm or cancel it.

Using the pixel events for Facebook ad optimization

Once you have nice pixels set up, you can use this in helping Facebook decide which versions of ads to serve, which audiences to serve them to, etc. You set up your ad, define an objective etc...

Define your goal as 'conversion', and define what 'conversion' corresponds to in terms of pixels:

Here we're choosing 'initiate checkout', which we defined as clicking on a 'create fundraiser' button on the first page of our site (early in the funnel)

The warning below might not matter as we

Facebook tracks people for a while. So in optimizing, you can change 'what time period of outcomes it attributes to which (version of the ad)':

I assume that the same 'conversions' target defined above is used in optimizing the 'dynamic creative' if you turn that on.

Example: "Hours spent promoting" vs "number of fellows"

Consider a study where

• EA groups are asked to voluntarily participate (with no direct compensation)

• to report the 'time spent on each recruiting activity',

• and to ask their fellows/members 'how did you hear about our group?'

Suppose this finds

'per hour spent by the organizers, far fewer people report "tabling" as the source, relative to 'a direct email'.

Should we interpret this as

'direct emails are a more efficient use of time than tabling, thus groups should spend less time doing tabling and more time sending emails?'

Maybe, but we should be careful; there are other explanations and interpretations we should delve into. Some of these could be partially addressed through survey design, others through careful analysis. Other 'causality' issues may require an experiment/trial/test to get at.

Statistical inference: chance and selection/selectivity

Random variation: With a small sample of groups, these numbers may be particularly high or low (for tabling, for emails, etc) by chance; the averages for a 'typical group' may turn out to be very different.

• This is the standard issue of statistical inference about a population from a sample.

• The issue of 'misrepresenting the population' tends to be worse with smaller samples (here small number of groups, and small numbers of observed outcomes in each group; e.g., only a few fellows)

• However, 'as Bayesians know' you can still draw valuable decision-relevant inferences from small samples. IMO (Reinstein) the "problem of small samples" tends to be overstated because we mainly learn about statistics designed for a particular scientific frequentist approach.

Selection/selectivity: The groups that 'opted in' to be part of this survey may not be a 'random draw' from the population of relevant groups. It may represent more careful or more enthusiastic groups, perhaps groups that are particularly analytical and not so good socially, etc. If some of the 'fellows' within the groups don't complete the survey, this could add another 'selection bias'.

'Marketing causality' issues

• Attribution with multiple sources: “How did you hear about this program?” This could be interpreted in several ways, probably “how did you first hear”. But in marketing sometimes people hear about something multiple times, and it’s hard to know which of these are pivotal in getting them to take action. (We could probably do something to make this question a bit more informative.)

• “Lift”: some people might have signed up anyways even without the activities identify as ‘how they heard about it’. Other people may have been harder to reach, and for the latter (e.g.) tabling ‘Spoke to us while we were tabling’ may be pivotal.

Decisionmaking implications

• Costs hours: the cost of these activities may not be fully proportional to the times spent … e.g., writing a professor may be mentally costly and possibly cost some other social capital. On the other hand tabling may be fun and social, and also generate interesting feedback (and other benefits that are harder to measure, like links with other groups also doing tabling)

• Diminishing returns/hard limits on some activities … e.g., there may be only so many professors (or students) to email. After a few hours of this

Barriers/obstacles to effective giving: classification

We focus on the 'barriers' or 'hurdles to giving effectively' among individuals who already engage in some charitable giving and other-regarding acts. Loosely, a donor would need to "jump over all of these hurdles" and cross each of these barriers in order to be giving effectively.

Conceptual breakdown (I)

A conceptual breakdown of barriers:

• Base values may be (non) utilitarian: People are optimising their own 'X', which does not coincide with impact --> no puzzle?

• Avoiding information about effectiveness: Even if people want to optimise impact, they may specifically dislike and avoid gathering information about effectiveness in a charitable giving setting

• Presenting effectiveness information may backfire: E.g., if it switches off the 'generous part of the brain', gets people to think in a more 'market' mode, or makes people indecisive

I present and discuss this breakdown, a more practical breakdown, and specific examples in each category in the part of the synthesis. I go into further detail an (work to) present evidence on each of these in later sections of that site. (I plan to go through that work and extract only the key, most practical elements).

These barriers are also mapped, and connected with tools and evidence in .

See 'barriers' View

Giving What We Can's mission is to make giving effectively and significantly a cultural norm. GWWC has updated their 2022 strategy. They are looking to significantly increase their marketing activity by producing videos, funding ads, and conducting systematic and robust research. As such there will be a large crossover between our work and theirs. This section highlights our collaborative efforts.

Presentation: overview

Ideas and opportunities

We want to learn from existing work, run tests on the GWWC platform, and support research into this.

Stages of the funnel:

1. Awareness & Consideration

   Increase casual visitors and raise curiosity

2. Conversion & Acquisition

   Donate or pledge to donate

Some key questions

• “What should the call to action be for the casual person in the funnel?”

• Testing all parts of funnel/pledge journey; website, welcome messages/welcome packages, reminders and thank-you's

Completed studies: See sections below

The presentation of options on GWWC's 'pledge page' were randomly varied at the individual browser level over a certain period to see which option increased pledges.

A summary of this has been shared as a post on the EA Forum.

Summary of trial and results

Giving What We Can (GWWC) has three giving pledge options, displayed in the 'Original presentation version' below.

From April-July 2021 they ran a trial presenting its 'pledge page' options in three slightly different ways. Considering 'clicks on any button' as the outcome, and a Bayesian 'preponderance of evidence' standard...

• "Separate Bullets for Other Pledges" was the most successful presentation. It only showed a box for "The Pledge", with the other options given in less prominent bullet points below. This had about a 20% higher incidence rate than the Original presentation.

• "Pledge before Try Giving" was the least successful presentation this was like the one displayed above, but with "Try Giving" in the central position. This had about a 23% lower incidence rate than the Original presentation.

These results may only apply narrowly to the GWWC pledge case, and even here, we have some . However, it loosely suggests that when making a call to action, it may be most effective to present the most well-known and expected option most prominently, and not to emphasize the range of choices (see further below).

Getting people to take the GWWC pledge may be seen as an important outcome on its own. It may have a causal impact on getting people engaged in the Effective Altruism community and other EA activities, such as EA career impact decisions.

General idea and main hypothesis

GWWC: How can we present pledge options to maximize positive outcomes (pledges, fulfillment)?

General: For those considering making substantial giving pledges (of a share of their income), how does the presentation of these 'pledge options' matter?

Theories and mechanisms to consider:

• Tendency to choose 'middle options'

• Too many options may lead to 'indecision paralysis'

• The signaling power of choice; e.g., if there's a 'more virtuous choice' I may feel that my 'middle choice' looks less good by comparison

Background and context

GWWC has three distinct pledge options, as shown

1. "Try Giving" (1% of income),

2. "The Pledge" (10% of income)

3. The "Further Pledge" (donate all income above a living allowance).

These can be seen on the 'pledge page' (link from October 2020).

Three versions of this page were randomly presented (between 19-21 April and 10 July 2021)

The content of the key 'choice button' part varied between these three versions

1. "Original:" A block of three (in the order of commitment) 'The Pledge' (10%) in the center and highlighted (see above)

2. "Pledge before TryGiving": A block of 3 with "Try Giving" (1%) in the center and highlighted

3. "Separate Bullets for Other Pledges": A single block for 'The Pledge' (10%), with the other pledges given as clickable bullet points below (as well as a bullet for the 'company pledge' ... which had a different presentation in other versions)

The version presented stayed constant according to an individual's IP cookie tracking.

Participant universe and sample size

• 'Everyone going to the above page' within the above time duration.

• People interested in GWWC pledges'

Sample size: see below, from Google Analytics

Key treatment(s)

1. "Original" (Block of 3 in order of commitment, Middle Pledge in Center)

2. "Pledge before TryGiving" ... as above but with Try Giving and The Pledge swapped, and Try Giving (in the center) highlighted

3. "Separate Bullets for Other Pledges" (see below)

Treatment assignment procedure

• Three versions of this page were randomly presented

• Equal likelihood of assignment

The version presented stayed constant for each individual across visits.

Outcome data

Statistics on Google Analytics: This records only 'pressed any button' (any pledge) as the successful outcome.

Ex-post: Reporting results (brief)

Implementation and data collection

See for details on data extraction from the interface

1. From shared image from Google Analytics:

'Experiment sessions' (observations) by treatment (as labeled on Google Analytics shared image):

Original: 2588

Pledge before Try Giving: 2686

Separate Bullets for Other Pledges: 2718

Total: 7992 sessions (=2588+2686+2718)

3. Where is the data stored ... [noted above]

Basic results/outcomes

Quick interpretation

The "separate bullets for other pledges" seems to have been the most successful, with an 0.49% higher (percentage point) incidence rate than the 'Original', i.e., a 22% higher rate of pledging (2.69 vs 2.20).

These differences seem unlikely to be statistically significant in a conventional sense. Still, Google analytics (presumably a reasonable Bayesian) model states an 80% chance that this is the best treatment, and this seems useful and informative.

The "Pledge Before Try giving" treatment performed substantially worse than the original.

These differences are meaningful–consider the 'posteriors':

Downloading the 'Analytics data' behind the above graphs, we see:

This suggests it is very reasonable to think that 'Separate Bullets' is substantially better

Our 'posterior' probability thus infers that we should put

• a 2.5% chance that 'Separate Bullets' (SB) has an 18% (or more) lower conversion rate than 'Original'

• a 22.5% chance on SB being between 18% worse and 4% better

• a 25% chance of SB being 4-20% better

We can also combine intervals, to make statements like ...

• a 50% chance of being 4-36% better

• a 50% chance of being 20-76% better

For 'Pledge before...' (PB) we can state, e.g.,

• PB has a 75% chance of being at least 11% worse than Original

• and a 50% chance of being at least 23% worse than Original

Intuitive interpretation

Perhaps giving people more options makes them indecisive. They may be particularly reluctant to choose a “relatively ambitious giving pledge” if a less ambitious option is highlighted.

This could also involve issues of self and social signaling. If the 'main thing' to do is a 10% pledge (as in "separate bullets"), then this may seem a straightforward way of conveying 'I am generous'. On the other hand, if the 'Further pledge' is fairly prominent, perhaps the signal feels less positive. And if the '1% pledge' is made central, 10% might seem more than a necessary signal.

The "pledge before try giving" may perform the worst because it makes the 'Try Giving' pledge a particularly salient alternative option. (In contrast, the "Original" at least makes 'The 10% Pledge' the central and the middle option.)

Caveats

I am assuming that the 'outcome being measured here' is whether the person 'clicked on any giving pledge'; this is what Luke has conveyed to me

I assume this is 'conversions ever from this IP', and 'sessions' represents 'how many different IPs came to the treatment'. If it's something else (e.g., each 'session' is a 'visit' from an individual), this could reflect these people converting in fewer sessions but not necessarily being more likely to convert overall. Even if this is 'by IP' the alternative interpretation 'not converting now but maybe later' may still have some weight if people are entering through multiple devices.

Introduction

Our , , , and the organization of

Five Key Sections

1) 🤝

In this section, you will find reports of the trials we have run with organizations, including Giving What We Can and One For the World.

2) 🪧

Here we share tools to implement planned trials, as well as tips relevant to 'doing marketing'. We answer questions like how to set up campaigns and track outcomes on various platforms. See especially "..." and "..."

3) 🎨

We discuss qualitative and quantitative research design and methodology issues that are relevant to the trials we are running. Pages in this section will be linked in reports when relevant to a particular trial.

4) 🧮

Our profiling project aims to help better understand what sorts of people are amenable to EA-related ideas and to taking EA-favored actions.

5) 📋

We've done a review of existing literature: to inform the trials we are running, and to identify important research topics. This includes and .

You can find references, tech support, and other resources in the .

What is this 'Gitbook' meant for?

The three key aims of this are to:

1. Convey who we are, what have accomplished, and the scope of our work to funders, people in the broader EA community, and people not yet involved in the project who would be interested in joining

2. Share tools and knowledge with people in the EA/global priorities community who will apply it to their work. We are building a knowledge base. Content in the public gitbook can inform and support a diverse set of projects (i.e., implementing marketing campaigns, fundraising initiatives, academic research)

See also

We recoded and augmented this analysis within the EAMT Analysis web-book here. More work could be done, if warranted.

Summary of trial and results

In December 2021, TLYCS ran a YouTube advertising campaign in Portland Oregon, involving ‘donation advice’. The top 10% household-income households were targeted with (one of) three categories of videos. One of the ultimate goals was to get households to sign up for a 'concierge' personal donor advising service

Quick takeaways

• There were very few signups for the concierge advising service. (About 16 in December 2021 , only 1 from Portland.)

• We consider a 'difference in difference', to compare the year-on-year changes in visits to TLYCS during this period for Portland vs other comparison cities.

• This comparison yields a 'middle estimate cost' of $37.7 per additional visitor to the site. This seems relatively expensive. We could look into this further to build a more careful model and consider statistical bounds, if such work was warranted.

General idea and main questions

Specific goal of TLYCS promotion: To get people to click on the ad and go to the 'landing page' of TLYCS. Here, they will fill out to request an appointment with a donation advisor. We will simultaneously be raising awareness for TLYCS.

General questions:

1. Can we get people to sign up for donation advice using videos in YouTube Ads?

   • How many sign-up and what sorts of people?

2. Do these ads boost engagement with TLYCS in net? (E.g. donations, website activity, book downloads)

Background and context

Participant universe and sample size

Location: Portland, OR

Audience: Top 10% of household income

People living in Portland, Oregon in the top 10% of household income (approximated by Google) will get an in-stream ad (ad plays before video user intended to watch)

Key treatment(s)

Exposure to a sequence of nine versions of YouTube ad videos. Frequency cap: 6/weeks

Three main 'theme/header' variations (similar, slightly different phrasings)

these variations were crossed with...

Three categories of videos within each theme:

1. "Bravery": Charlie Bresler explains how 'you can save lives without being brave' with small amounts of money for bednets, nutrient micro-doses, etc.

2. $10: Man giving out money to poverty-stricken people in Capetown. Text narrative overlaid describes that $5 can buy a slice of pizza, or an interocular lens to treat cataracts, etc. Leans towards 'identified victims/recipients'.

3. "I want to do good": Colorful puppets sing about giving and donating to save lives. Counters common arguments about 'breeding dependency', fear of administrative waste, etc.

These are organized and linked .

Each video comes with a site-link extension with a Call to Action:

Treatment assignment procedure

We assigned the particular video treatments to audiences using a YouTube/Google optimization algorithm. This chose videos to maximize the probability that a user chose 'Speak to an Advisor' and filled out the linked form.

Outcome data

• How long people watched the videos for

• Whether they 'clicked through'

• Whether they filled out the form for advising (Algorithm is serving to optimize this)

Results (simple analysis)

Cost per user (first-pass)

A first pass and upper bound on impact and (lower bound on) cost/session

Assumptions/data interpretations

1. The numbers used in our data come from meaningful sessions from unique users

2. The 'date range' is the relevant one for being affected by the advertisements of interest

3. The 'comparison cities' are approximately randomly selected

Most optimistic (unrealistic) bound

Guiding assumption: a counterfactual 0 visits from Portland in this season

• 306 Portland Users (389 Portland site visits) in relevant 2021 period.

• If these were all driven by the advertisement (and counterfactual was 0 visits), this is +306 Users and +389 visits

• Cost $4k

Year-on-Year (maybe reasonable) optimistic bound

Guiding assumption: a counterfactual 'sam as last year' in Portland

• 306 Portland Users (389 Portland site visits) in relevant 2021 period.

• 144 Portland Users (189 Portland site visits) in relevant 2020 period.

• --> 306 - 144 =162 users uptick,

Difference in Differences comparison to other cities

Guiding assumptions:

• The cities used are fairly representative

• 'Uptick as a percentage' is unrelated to city size/visits last year

• All the cities in the comparison group are 'informative to the counterfactual' in proportion to their total number of sessions

This yields

• 112.5% visits uptick (Year on Year) for Portland in 2020

For all North American cities other than Portland (with greater than 250 000 people):

The average is 46.5 users in the 2020 period and 64.5 users in the 2021 period, an uptick of about 38.8%. This is very similar to the result if we look at all cities which has an uptick of 43.1%.

• 38.8% uptick multiplied by 144 users = 55.9 (‘counterfactual uptick’ in users for Portland)

• 162 - 55.9 = 106 (uptick relative to counterfactual)

• USD 4000 /106 = 37.7 USD cost per additional user through this ad

Note this is a midpoint estimate, we have not yet given statistical bounds.

In the graph below (pasted from the Quarto ), we show these year-on-year upticks in context.

Other outcomes

There were very few signups for the concierge advising service. Only about 16 in December 2021 globally, only 1 of which was from Portland.

Notes

Other detailed notes are in our private Gitbook. More formal and detailed analysis could be done if it seems merited.

Along with GWWC, we tested marketing and messaging themes on Facebook in their Effective Giving Guide Facebook Lead campaigns. Across four trials we compared the effectiveness of different types of (1) messages, (2) videos, and (3) targeted audiences.

A summary of this has been shared as a post on the EA Forum:

We build the results and analysis transparently in the EAMT Analysis web-book here.

Summary

Context: Facebook ads on a range of audiences

... [with text and rich content promoting effective giving and a "giving guide" -- links people to asking for their email in exchange for the guide]

Objective: Test distinct approaches to messaging, aiming to get people to download our Giving Guide. A key comparison: "Charity research facts" vs. "cause focus".

Also informative about costs and the 'value of targeting different groups' in this context.

Key findings:

• The cost of an email address via a Facebook campaign during Giving Season was as low as $8.00 across campaigns.

• “Only 3% of people give effectively,” seems to be an effective message for generating link clicks and email addresses, relative to the other messages.

Key caveats

Specificity and interpretation: All comparisons are not for 'audiences of similar composition' but for 'the best audience Facebook could find to show the ads, within each group, according to its algorithm'. Thus, differences in performance may combine 'better targeting' with 'better performance on the targeted group'. See our . I.e., we can make statements about "what works better on Facebook in this context and maybe similar contexts", but not about "which audience, as defined, is more receptive", as the targeting within each audience may differ in unobserved ways.

The outcome is 'click to download the giving guide'.

Academic-linked authors: David Reinstein, Josh Lewis, potentially others going forward

Implementation and management: Chloe Cudaback, Jack Lewars

AsPredicted questions

1) Have any data been collected for this study already?

No, no data have been collected for this study yet.

2) What's the main question being asked or hypothesis being tested in this study?

Are effectiveness-minded (EA-adjacent) donors and pledgers more motivated to donate by

1. "A": A (non-quantitative) mention of impact and effectiveness (in line with the standard OftW pitch)

2. "B": Emotional appeals and 'identified victim' images

Framing this in terms of the psychology, social science, and philanthropy literature:

"Does the Identifiable Victims Effect (see e.g., meta-analysis by Lee and Feeley, 2016) also motivate the most analytical and committed donors?"

3) Describe the key dependent variable(s) specifying how they will be measured.

• d_don_specific: Whether the person receiving the series of emails makes an additional 'one time gift' following the link at OftW, within the OftW interface, during the 'Giving Season', a time-period that (for this preregistration) we declare to begin on receipt of this first email and end on 15 January 2022.

• don_specific: The total amount donated through the above

4) How many and which conditions will participants be assigned to?

Two conditions (treatments):

A. "Impact"

B. "Story/Emotion"

Assignment details

Participants (c 4000 people at various points in the One for the World pledge process) will be split into groups (blocks) by previous donation behavior or point in the process. (OftW have mentioned, pledgers still in school, active donors, and lapsed donors).

Within each group, they will be randomized (selection without replacement to ensure close-to-exact shares) into equal shares in treatments A and B.

Treatment specifics (i.e., 'experimental conditions')

A series of three emails will be sent, with participants remaining in the same treatment across all three emails.

See actual texts for design and timing

Example content differences, from email 1:

A. Impact version:

As of 2021, One for the World has had a tremendous impact on the lives of those that are helped by our charity Top Picks programs:

[IMPACT SINCE 2021 GRAPHIC]

B. Story/Emotion version:

Here’s our first story this season from Eunice of Kenya. When asked how her life changed when she received the first cash transfer from our partner organization, GiveDirectly, she responded”

“I have been able to make new goals and achieve them since I started receiving this money [from GiveDirectly]. I have been able to buy a piece of land that would have taken [me] many years to earn [enough to buy the land]. I was also able to buy livestock, like goats. I have even managed to dress my family properly by buying them decent clothing. Lastly, I have even been able to [pay my children’s] school fees without any strain.” (Source GiveDirectlyLive)

[PICTURE OF EUNICE]

5) Specify exactly which analyses you will conduct to examine the main question/hypothesis.

We will report all of the following analyses, with our preferred method in bold:

Binary outcomes:

• Fisher's exact test

• Bayesian Test of Difference in Proportions (as in ), with an informative beta distribution for the prior over the incidence rate in each treatment, with a parameter based on the incidence rates for similar campaigns in the prior 2 years.

Continuous outcomes:

• Standard rank-sum tests (Mann–Whitney U test)

• Simulation/permutation based tests for whether the mean (including 0's) is higher in group A or B (including 0's)

• ... same for median, but medians will almost always be 0, we anticipate

All tests will be 2-sided.

We will also report Bayesian credible intervals and other Bayesian measures for the proportion tests. We may also explore Bayesian approaches for the continuous outcomes, e.g., Bayesian beta regression.

We also anticipate reporting multiple-hypothesis-test corrections, but we are not pre-registering a method. Our approach to this is likely to follow that of List et al (2017), which this paper applied to a similar domain (charitable giving experiments with multiple donation-related outcomes).

We will report confidence intervals on our results as well as Bayesian credible intervals under flat and weakly informative priors. Where we have a 'near-zero' result, we will try to put reasonable bounds on it to convey the extent of our certainty that the true effect or parameter was fairly small.

Where situations arise that have not been anticipated in our preregistration and pre-analysis plan, we will try to follow the Don Green lab unless there is a very strong reason to deviate from this, which we will specify.

6) Describe exactly how outliers will be defined and handled, and your precise rule(s) for excluding observations.

• Included: All individuals who received this mailing.

\

We will not exclude any observations from the sample, unless they make it clear to us that they are aware of this trial.

We will not Windsorise or exclude outliers.

7) How many observations will be collected or what will determine sample size?

A series of three campaign emails will be sent out by OftW to their regular email lists, to roughly 4000 participants, as described above

Targeted dates: November 10, November 18, November 23, all in 2021, but these may be delayed for feasibility

Other

Anything else you would like to pre-register? (e.g., secondary analyses, variables collected for exploratory purposes, unusual analyses planned?)

Exploratory and secondary hypotheses/questions/analyses

Secondary hypotheses and questions

Which treatment motivates a higher rate of...

• Email open rates (note, as we have three obs per participant, we will need random effects or clustered standard errors). and

• Use click rates (with same caveat)?

We consider these as secondary because the click and open rates do not necessarily strongly relate to outcomes of interest, particular among this set of already effectiveness-minded donors. These outcomes may simply reflect attention or curiosity about the content.

\

Exploratory: what factors (especially gender, university/student status, university subject) predict which treatment leads to greater donation (incidence and amount)

Note that our partner is planning to use this trial to inform future trials and experiments, particular for the 'Giving Tuesday' season itself.

Power calculations

We did not have time to do even simple power calculations before the start date of this experiment. However, we will try to conduct these before we obtain any of the data, and update this preregistration.

We are a group of researchers and practitioners across a range of fields (Economics, Psychology, Marketing, Statistics) and organizations, particularly those interested in effective charitable giving and effective altruism. This is outlined in the Airtable (invite link), with embedded views below.

This project is organized by David Reinstein, who maintains this wiki/Gitbook and other resources.

As individuals and organizations, we are goal-driven and impact-driven: we are in this to improve the world, particularly through directing funds and support to the most effective rercauses and interventions. Because we share these common goals, we are better aligned for collaboration than typical academics and charitable organizations. We have an unprecedented opportunity to collaborate, learn what works, and 'move the needle'.

We are actively collaborating with the following organizations (links indicate publicly reportable trials)

Background on OftW

• OftW has a donor base of ~700 active donors, ~1650 pledged donors (who pledged but haven't started donating yet) and ~2000 lapsed donors.

• 80% (of donors?) are in the USA

• Focus on global health charities

Some key goals

Reinstein/Lewars conversation notes

• Activating more donors who took the pledge at university, so their donations actually start;

• Retaining donors for longer once they activate;

• Upselling donors to give more over time (either more as a raw amount, e.g. 'keeping pace' at 1% of their income; or more as a percentage, e.g. 'graduating' to take the 10% GWWC pledge)

Who/what/how to test, learn, and adapt

Pipeline/groups/segments

1. Pledgers

2. Active donors, i.e., "Activated pledgers" (Chloe is thinking of segments to this and how to appeal to them)

   1. Second tier -- people who have given each month for 12+ months; "Legacy donors" (DR: maybe 1x per year high-value donors should be in this group)

• Another group worth considering: 'pledge-curious supporters'

Goals/actions

• 'Activating' Pledgers as donors (pledged but not donated)

• Active donors

  • Retain

Interested in knowing more about

• Content -- expand our ability to tell stories about the beneficiaries

  • Ways to tell these stories

• Frequency (of comms with supporters)

Communications contexts

• Platforms: Social media, email flows

• Telling stories in a corporate context

Typical audiences have been students and young professionals, but there is interest in corporate outreach

• Zoom and lunchtime talks in corporate contexts (How many? Seems very promising!)

  • How many people are activating/pledging following these lunch+learn?

Typical donor journey:

We are in the process of creating these homepages and setting up conversion tracking. As OFTW has ~0 organic sign ups currently, we are testing for a variety of conversion routes, including: [Todo: clarify this]

• university campus, someone I like tells me they are involved in OftW, asks me to come along with free food

• at some point I take the pledge

• It is not a highly controlled process

Some rough numbers

650 active donors

1500 people in pipeline (pre-activation date)

750 new people a year are recruited... thinks it would be 2-2.5k

OFTW has a donor base of ~700 active donors,

~1650 pledged donors (who pledged but haven't started donating yet) and

~2000 lapsed donors.

Ongoing/completed/upcoming experiments

Email upsell emotion/impact message trial (see below)

University experiment - redacted as being prepared

• Homepage message testing

• Activation trial

Summary

Main Question: Do some message themes work better than others for drawing visitors to Giving What We Can’s landing page?

Main findings: 'Social proof messages' on Facebook ads were most effective at generating landing page views per dollar compared to other message themes (effectiveness, services, giving more, and values).

Future directions: There were significant differences in 'link clicks per dollar' on the different messages by age. We recommend a systematic test to determine if age makes a difference in the relative effectiveness of social proof and values messages. Future studies could explore why the social proof message was more effective in this study than the previous giving guide study and the importance of the message to “join” the movement as social proof.

Possible connection between this trial and the : Note that the two best-performing messages both prompted the user to “join” a movement or a group of people (perhaps an elite group); but beware .

to report below.

Pre-trial reporting template

General idea, main 'hypothesis' (if there is one)

In this test, we are aiming to find out if one 'theme' of messages resonates better with our target audience than others.

If we knew which 'themes' were most effective with our advertising, then we could create more ads on this theme and improve our conversion.

Specifically, which of the following themes resonate with our target audience the most:

• effectiveness

• giving more

• social proof

On choosing an objective of this test, originally I planned to use link clicks, but this is not the most high quality indicator of conversion, and when I tried to use newsletter signups Facebook warned me that I might not see any conversions at all... So instead, the campaign will optimise for landing page views, which is slightly better than a link click and will generate enough conversions that we should [see?] we statistically significant results.

Point of contact (at organization running trial)

Grace Adams

Timing of trial (when will it/did it start and end, if known)

Trial will run for 7 days on GWWC's ad account, from 9.30am AEDT Friday 25 Feb to 9.30am AEDT Friday 4 Mar.

Digital location where project 'lives' (planning, material, data)

Working document can be found but all important details will be listed in this brief

Environment/context for trial

This test will take place on Meta platforms including Facebook and Instagram

Participant universe and sample size

• We are targeting a "Philanthropy and Lookalikes (18-39)" audiences, based in UK, US or Netherland

• Estimates from Facebook: Reach is expected to be 1.4K-4.1K per day (7 days) per ad set (5 ad sets) = 49K-143K

• Estimates from Facebook: Conversion is expected to be 10-30 landing page views per day (7 days) per ad set (5 ad sets) = 350-1050

Key treatment(s)

We are using the GWWC Brand Video by Hypercube as the creative across all tests. Although it did not perform as well as our other ads in the Giving Guide campaign, I think that it will interfere less with our messages we aim to test.

We are going to test a set of messages for each theme, please see them in the

Mock up of ad:

Treatment assignment procedure

• This test has been set up as an A/B test through Facebook, testing each campaign head to head, each campaign covers one theme, with the different ads as a child.

• This will allow us to test which theme was better, not just which individual ads

• A/B testing on facebook will ensure that the audiences fall into an individual treatment group

Outcome data

• Primary measure will be cost per landing page view, but secondary measures such as CPC, 3 second video plays, email sign ups will also be tracked

• Data will live on Meta ads platform

In the Partner Organizations and Trials section, you will find reports of the trials we have run with organizations, including and .

These trials are also cataloged in our Airtable: (that is...); links, categorization provided.

Our primary approach and goals

We want to identify the most effective and scalable strategies for marketing EA and EA-adjacent ideas and actions. To do this, we believe that running real-world marketing trials and experiments with EA-aligned organizations will provide the best evidence to act upon. By systematically varying the messaging, framing, and contexts, we can map out 'what works better where'.

We believe this approach is likely to be the most fruitful because:

1. Using naturally-occurring populations in real-world settings with meaningful costly choices and outcomes will lead to more relevant findings. In comparison to convenience samples of undergraduates or professional survey participants who are aware that they are doing a research study, we anticipate greater:

   • Internal validity: our results are less likely to be influenced by biases, such as acquiescence bias and hypothetical decision-making.

Key themes, priorities, and 'high-value questions'

This project primarily aims at:

1. Robust and generalizable insights that improve communication and messaging

2. Meaningful and relevant long-run outcomes, such as;

   • Creating new, strong EAs by getting people more interested and involved in EA ideas, actions, and the community

In the document below (), we consider the shared goals, paths, and questions that are valid across organizations. Specifically, these are actionable and promising themes and projects that can be implemented, measured, and communicated fluidly throughout the EA network.

Context: Donation 'upsell' to existing pledgers

Question: Are effectiveness-minded (EA-adjacent) donors and pledgers more motivated to donate by

1. "A": (non-quantitative) presentation of impact and effectiveness (as in standard OftW pitch)

2. "B": Emotional appeals and 'identified victim' images

General idea, main 'hypothesis'

Are effectiveness-minded (EA-adjacent) donors and pledgers more motivated to donate by

1. "A": (non-quantitative) presentation of impact and effectiveness (as in standard OftW pitch)

2. "B": Emotional appeals and 'identified victim' images

In the context of One for The World's (OFTW) 'giving season upselling campaign', potentially generalizable to other contexts.

Academic framing: "Does the Identifiable Victims Effect (see e.g., the meta-analysis by Lee and Feeley, 2016) also motivate the most analytical and committed donors?"

Background and context

One for The World's (OFTW) 'giving season upselling campaign''

10 emails total over the course of November were sent in preparation for GivingTuesday

Point of contact (at organization running trial)

Academic-linked authors: David Reinstein, Josh Lewis, and potentially others

Timing of trial

Targeted dates: November 10, 18, 23, all in 2021, but may be delayed for feasibility

Digital location where project 'lives' (planning, material, data)

Present Gitbook, Google doc linked below, preregistration (OSF), and github/git repo

Environment/context for trial

Emails ... to existing OftW pledgers (asking for additional donations in Giving Season)

All 10 emails had the same CTA: make an additional $100 donation for the giving season/GivingTuesday on top of their recurring monthly pledge donation.

Participant universe and sample size

Roughly 4000 participants, as described.

A series of three campaign emails will be sent out by OftW to their regular email lists, to roughly 4000 participants, as described.

Key treatment(s)

Basically:

• A list of ~4500 contacts (activated pledgers) was split into two treatment groups.

• Treatment Group A received emails that were focused on the contact's impact

• while Treatment Group B received emails that were focused on individual stories of beneficiaries

See

Treatment assignment procedure

See preregistration

Outcome data

Targeting: Donation incidence and amount in the relevant 'giving season' and over the next year, specifically described in prereg under

Data storage/form:

• MailChimp data (Chloe is sharing this),

• Reports on donations (Kennan is gathering this)

Optional/suggested additions

Planned analysis methods, preregistration link

Cost of running trial/promotion: Time costs only (as far as I know)

Proposed/implementing design (language)

(

Pre-registration work

Pre-registered on OSF in 'AsPredicted' format, content incorporated here

Preliminary results

Overview:

The Emotion treatment leads to significantly fewer people opening emails, but more people clicking on the in-email donation link (relative to the standard Impact information treatment). However, we are statistically underpowered to detect a difference in actual donations. More evidence is needed.

Chloe: those emails that appealed to emotional storytelling performed better (higher in-email click rate) than those that were impact-focused.

DR, update: I confirm that this is indeed the case, and this is statistically significant in further analysis.

Evidence on donations

(preliminary; we are awaiting further donations in the giving season) ...

Treatment 1 (Impact): We record

• 1405 unique emails listed as opening a ‘control’ treatment email

• 29 members clicking on the donation link in an email at least once (2.1% of openers)

• 15 members making some one-time donation in this period (about 0.11% of openers, 0.075% of total)

Treatment 2 (Emotional storytelling):

• 1190 unique emails listed as opening an email (a significantly lower 'open rate', assuming the same shares of members were sent each set of treatment email)

• 56 members clicking on the donation link in an email at least once (4.7% of openers)

• 11 members making some one-time donation in this period (about 0.9% of openers, about 0.055% of total)

‘Initial impressions of preliminary outcomes’

• The conversion rates are rather low (0.5%) … but maybe high enough to justify sending these emails? I’m not sure.

• While people are more likely to O_pen_ at least one Impact email, they are more likely to Click to donate at least once if assigned the Emotion email

• But we can't say much for actual donations.

The figure above seems like a good summary of the ‘results so far’ on ‘what we can infer about relative incidence rates’, presuming I understand the situation correctly …I plot Y-axis: ’how likely would a difference in donations ‘as small or smaller in magnitude’” than we see in the data between the incidence … against X-axis: if the “true difference in incidence rates” were of these magnitudes

Implementation and management: Chloe Cudaback, Jack Lewars

• Our data is consistent with ‘no difference’ (of course) … but it's also consistent with ‘a fairly large difference in incidence’

• E.g., even if one treatment truly lead to ‘twice as many donations as the other’, we still have a 33% chance or so of seeing a difference as small as the one we see

• We can reasonably ‘rule out’ differences of maybe 2.5x or greater

We have completed various trials in conjunction with , the most recent being the city-level YouTube test. There are a number of additional proposed trials and tests, however, at the moment these considerations are limited to the private Gitbook.

Note that in the past TLYCS has worked with the Graduate Policy Workshop School of Public and International Affairs at Princeton University, who produced the report embedded below.

https://app.gitbook.com/u/WrM9GjKWCyRyoIjCKt7f0ddJwCr1's research (along with others) considers 'how do potential donors respond to (different presentations of) impact information'. Reinstein and his academic partners ran several experiments, working with (and on) mainstream charities and fundraising platforms.

See work:

and discussion:

Other work is ongoing and cannot be publicly shared yet (see private gitbook if you have access).

See also the cross-organization notes on advertising, google, youtube, etc (=placeholder for now) and the tips on Doing and funding ads

YouTube Remarketing

July 20, 2021: GWWC launched a YouTube remarketing campaign. That means that when someone goes to the GWWC website, leaves, and then goes to YouTube we show them one of the following videos:

Algorithm decides which video to present to people.

Understanding assignment, proposing experimental design ’s questions:

Q: Is each video assigned to a different situation or are videos randomly chosen to be displayed? If the latter, you could randomize videos by location and see if the different videos were more or less effective. Alternatively, just randomizing the whole campaign seems like a good idea to me....

A: Videos are selected based on the likelihood of the user watching >30 seconds (by the algorithm) ... randomization by individual will be hard because users don't click and act right away. Instead I think we have to randomize by geography

Results summary (Early, JS Winchell; may need update)

Most important takeaway: It costs $1 to get a website visitor to watch 1h of your videos! High level metrics

• Cost: $205

• Views: 6,071 (a view is when a user chooses to watch >30s of an ad)

• Total watch time: 223 hours (~$1/h)

Interesting observations\

1. Efficiency has significantly improved over 3 weeks

• Cost per view has gone down from $0.05 per >30s view to $0.02 per >30s view

• Views have increased 75% without increasing budget (from 220/day to start to 386 yesterday)

2. 10% of the time people watched the full video! \

• You can see this data by video if you are interested to control for video length

• E.g., 5% of people chose to watch the entire 13 minutes of _

3. Your best video had a view rate (% of time people choose to watch >30s) twice as good as your worst video 4. You can see view rate by age, gender, and device in the "Analytics" tab

• For the , older people and men were more likely to choose to continue watching

Possible next steps

• Could add "similar audiences" which is when we let Google use machine learning to find people similar to your website visitors and also show ads to them

• Could walk David Reinstein and Joshua Lewis through the UI so they can get a sense of the metrics/reporting available and how it could be used for research

EA seeks to amplify its impact through movement-building. Organizations like 80,000 Hours and CEA are putting substantial resources into developing and expanding the EA community. Building EA groups has been at the core of this agenda, especially in elite and influential places (such as top universities). Key aims include 'creating highly engaged EAs' and encouraging people to pursue impactful careers.

Resources and tips

(in-progress)

Our collaborative goals

Currently, university EA groups operate in conjunction with the Centre for Effective Altruism, but with high levels of autonomy. There is only limited collaboration between groups. Such collaboration could allow them to achieve economies of scale and scope, run more systematic and powerful trials, and share insights and methods that increase student engagement.

The EAMT hopes to help coordinate this, consolidate the evidence, and provide accessible tools to newly-formed groups. We want to help avoid repeating errors and 'reinventing the wheel' each time.

EA groups at universities

The efforts and experience of individual EA groups can provide contextual evidence and insights. The EAMT aims to aggregate this knowledge, find generalizable principles, and disseminate this to the wider EA community. We are focused on meaningful medium-term outcomes, e.g.:

1. Membership and participation in EA organizations, and markers of post-university involvement

2. How career plans are impacted (focusing on particular programs and paths)

3. How research and discourse at universities can be influenced

Relevant organizations and programs

The programs below also aim for generalizable principles; e.g., their 'starter toolkits' are implemented across a range of cities, universities, and settings.

Centre for Effective Altruism

CEA has , passing funding and efforts on to Open Philanthropy. However, CEA is still involved in promotion through the (UGAP), which offers guidance and resources to newly formed groups. Furthermore, CEA's (CBG) Program helps develop national and city-based groups (outside of universities).

University Group Accelerator Program

may be the best current source of centralized knowledge for approaches to outreach methods. These have been summarized from different data points; some formal testing, some anecdotal, and some intuitive.

Community Building Grants Program

CBG focuses on supporting city groups, providing grants to support their activities and resources to help with expansion. These resources and support systems currently lack data supporting EA community building. (The identified this as a major bottleneck; we hope to collaborate to help them improve this.)

EA Group Organizers Survey

The is a collaboration between CEA and Rethink Priorities. It analyzes the changes in EA groups yearly, with two main components:

1. The growth and composition of EA groups and their activities

2. The opinions of the group's status from the organizer's point of view

The first component gives insight into priorities and progress. The second can help guide our research and provide insight into the tools required by group organizers to increase group interaction and outreach.

See especially:

Open Philanthropy

provides funding for part-time and full-time organizers helping with student groups focused on effective altruism, longtermism, rationality, or other relevant topics at any university (not just focus universities). This has replaced CEA's Campus Specialist and Campus Specialist Internship programs.

, a selective 2-year program that gives resources and support (including $100K+/year in funding) to particularly promising people early in their careers who want to work in areas that could improve the long-term future. (Intended partially for particularly strong Campus Specialist applicants.)

80,000 Hours

80,000 Hours is actively targeting university students and offering them guidance on high-impact career paths. (see private Gitbook, if you have access)

Further outreach

There are some further initiatives in this area but most of the material cannot be shared at the moment (see private Gitbook).

Independent group testing and coordination

In this section, we are putting together documents, trials, and knowledge currently being gathered by different EA groups. As we increase our collaboration with these groups, these trials, ideas and documents will become integrated with the Gitbook and EAMT's work, forming a basis for future work and testing.

Funnel Map

This is our basic understanding of the processes used to draw in new members to EA university groups and fellowships, and how members progress through different stages of engagement. Each stage gives us grounds for testing through the different variations of these approaches. This is not just about testing which methods work for attracting the highest number of new members (i.e., which 'call to action' to use at activity fairs, etc), but also increasing engagement and developing high-level EAs (i.e., fellowship program alternatives, discussion group topics, etc).

(Above: a preview of funnel map; for full description and work in progress)

Stanford University

Awaiting response from Stanford EA.

University of Chicago

Currently limited to private Gitbook.

MIT, MIT Alignment org

Currently limited to private Gitbook.

EA Israel

This discusses their strategy in-depth. A lot of their findings are not specific to Israel or country-wide EA groups. Useful as a resource for EA groups.

Useful findings will be synthesised and integrated here in the future.

Independent University Group Outreach

We have been independently contacting organizers that are known to be actively seeking to test outreach methods, and also publicly via a on the EA Forum. An important aspect of the work here is to bring together people who are active in this space but working independently. The airtable below presents our current (non-exhaustive) list of groups or organisations that have relevant knowledge (strategy documents, marketing guides, etc), or have done some form of independent testing.

1. Thus, we hope our efforts will be valuable to these initiatives and groups, by providing and sharing evidence on successful approaches to increasing engagement.

2. Note that the survey does not collect data from the group's *members*, although they do ask about the overall numbers of people who engaged with each group.

What's been/being done, what do we know?

Discussion space

In case you don't like writing in this Gitbook, I created

EA Forum posts

See the Google doc embedded below HERE and feel free to add comments or questions.

Note 7 Mar 2023: I just started this page, it is far from complete

Organizations and 'do' initiatives

: Including (new) EA-aligned marketing groups

Why use survey/experiment/marketing platforms?

• To test content in more depth than an A/B trial permits

• Better control over 'who is participating' and how much attention they are paying

• Things more towards 'basic background research'

• Closer to a 'representative sample'

Some participant recruitment platforms

: Created specifically for (academic research). Our impression is that this is among the highest quality panels, although there is some controversy over this.

CloudResearch: CR approved Mturk

CloudResearch: Prime Panels

Positly: https://www.positly.com/ Qualtrics (panel) Lucid

Dyndata

Contexts to test outreach messages

Contexts allowing individual randomization & tracking of medium-term outcomes

1. GWWC web site at point of email signup

2. Email lists

   • immediate: subject headers w/ 'open rates' as dependent variable

Contexts for 'Immediate outcomes' (clicks etc)

Facebook; But the targeting algorithm may frustrate randomization. (see .) Can it be switched off?

Contexts allowing

See

This is helpful if the important outcomes can be tracked by ZIP code/post code/address.

• Online display advertising

• Google search

• YouTube

Testing Rich Content

How to test rich content?

We can use some of the same strategies as above to test "rich content", i.e., short or even long talks, book chapters, podcasts, and so forth.

Targeting

See also

Question: If our aim is to change the culture of giving in general, what kind of people should we be targeting?

1. Influencers (People with lots of social influence)

2. Low-hanging fruit (i.e., people who are naturally predisposed towards effective giving, pledging, & EA)

Ideas/methods for targeting: platforms and audiences

Cost of ads: benchmarks

Reinstein, FB ads tied to fundraisers.

My costs have been:\

• about $0.01 per impression

• about $0.50 - $1.20 per click

Targeting at Universities ... Facebook's estimates

The estimated cost per impression (?‘reach’) and per click varies with the targeted audience. In general, narrower is estimated to be more costly. I think this is about ‘a larger audience allows FB to serve the ads to a larger number of people who tend to be click-happy’ Some data points:

• For Oxford, ‘In College’, living in the UK: They claim we will get 4-18 clicks per day for $50 per day over 2 days (29 Mar 2022 check on FB ads manager)

• If I put in Birmingham instead I get a fairly similar figure.

• If I remove the only-one-university narrowing, it gets cheaper. They claim I’ll be able to get 86-250 clicks per day for the same cost …

Meta allies

Research advancement manager:

EA groups (employees) within Meta

If you run a lot of ads FB will assign you external consultant helpers. They are somewhat helpful, but they don't seem to know everything.

FB tips for charities/fundraisers

28 Nov 2022, Zoroob:

Meta has just released a recorded series of videos () to help non-profit organizations meet their year-end fundraising goals. Some of these materials may also be helpful for researchers using Meta ads (e.g., materials on designing effective ad creatives), so I am passing the info along. Blurb below.

The three-part series of virtual webinars provides nonprofits with advertising training and best practices around how to use Meta technologies to further their missions:

: Get started with Meta advertising with our today.

The session also features that enable donation transactions within the Facebook app.

: Learn what great nonprofit creative can look like with best practices from Meta.

Consider saving our to learn more about the five key creative considerations that apply to cause-driven campaigns.

: Introduce yourself to measurement best practices on Facebook and Instagram! Afterwards, explore split testing, lift measurement, and the experiments tool, on our

Other links and issues

(collecting data)

Seems particularly useful but access is limited; they hope to make it more generally available some thing like mid- spring 2023.

Google ads grants

JS Winchell presentation:

Tomasik (2014) post

General grants -- see "" section

For a trial to yield insight, we need to be able to track and measure meaningful outcomes, and connect these to the particular 'arm' of the trial the person saw ... (if they saw any arm at all)

Notes from conversations (need explanation)

In this section we discuss how to see the results of your promotions and trials, and how to access data sets of these results that you can analyze.

• and

• and

Pivot tables

You may want to see or export crosstabs of one outcome, user feature, or design feature, by another. Sometimes you just want to see these quickly, but this might also be a way to extract the 'raw data' you wish to analyze elsewhere.

1. Start new pivot table

From within Ads Manager From 'ads reporting' (3 Aug 2022 updated interface)

Click "Create Report" --> Pivot table

2. As before, make sure you've selected the right date range, and (redo) any relevant filters

Here I add a filter for 'campaign name ' contains 'general'. Because I'm specifically trying to pull down some information on 'which video people saw' in this group (which needs a special setting to access... as noted below)

3. "Customize pivot table" – "Breakdowns" ... the things you want this to disaggregate across (sums and averages within groups)

• the 'campaigns', the 'ad names'

• timing, demographics

• Drill down to "Custom breakdowns", "Dynamic Creative Asset", to get it broken down by the text linked to the ads:

4. "Customize pivot table" – "Metrics"

Select the things you want reported, and deselect things that are not interesting or irrelevant to this case (like 'website purchases') or numbers that can be easily computed on your own

Normally, I'd suggest leaving out the redundant 'Cost per Result' but it's probably good to have as at least one sanity and data check.

Other stuff like 'video play time' could sometimes be very relevant, but I'll leave it out for now

I added a few features I thought might be interesting or useful. Was anyone drawn in to pledge? When did each campaign start/end (doublecheck)? How many unique link clicks?

5. (Optional) Conditional formatting

This could also be helpful if you are using the Ads Manager tools in situ, but obviously this has no value for downloading.

6. Save report for later use, share

If you think the report is useful in-situ, you can also share a link

7. Export the data

As in ...

(or consider direct import into R using tools like the rfacebookstat package)

Sections

"Qualitative" design issues: How to design the 'content' of experiments and surveys to have internal validity and external generalizability

Real-world assignment & inference: How to set up trials to have comparable groups

Adaptive design/sampling, reinforcement learning: Adjusting the treatments and design as you learn, to 'get to the highest value in the end'

: How to make inferences from the data after you have it (and plan this in advance)

What are our estimation goals?

Statistical power versus optimized learning

Fixed vs adaptive designs

See

Resources

Rethink Priorities notes (some are works in progress)...

The framework and R package seems very helpful. I (David Reinstein) am learning and trying to adapt it.

Mapping the key non-obvious features of running and analyzing these A/B trials using the Google analytics/optimize system.

Reporting and considering this in the context of the GWWC Pledge page (options trial)

What is it? How to create a trial

Setup and requirements

Set 'variants' and give them weights (assign shares for each)

Page and audience targeting

Objectives: The things you are trying to measure and improve

"Activity": Trials created/started/changed/ended/archive

Viewing results and Google's analysis

Clicking on a particular 'experience' in the 'container'...

(if you have been granted read and analyze permission), will open the useful 'Optimize Report' (which Google explains )

Optimize report: top

The overall start/end and 'sessions' are given first. What are "sessions"? The short answer: 'Sessions' are the number of 'continuously active' periods of an individual user. So individual users may have multiple sessions! (see below). Here, there have been 7992 such 'sessions' over 81 days.

• I am not sure where we can learn 'how many users there were'.

• ("View full chart" can give you a day-by-day breakdown of the number of sessions.)

OR: Conversion rates section

The next section compares 'sessions' and 'conversions' by treatment, and does a Bayesian analysis. This seems the most useful part:

Relative conversion rates, analysis

Above, the 'Separate block' (SB) seems to be the best performing treatment. Google calculates a 2.69% conversion rate for this (here, presumably the rate of people checking 'any' of the follow-on boxes).

Considering the Analysis, Google Optimize "uses Bayesian inference to generate its reports... [and] chooses its priors to be quite uninformed." The exact priors are not specified (we should try to clarify this).

But if we take this seriously, we might say something like ...

if our initial priors gave each treatment an equal chance of having the highest conversion rate ('being best'), and assumed a [?beta] distributed conversion rate for each, centered at the overall mean conversion rate ...

then, ex-post, our posterior should be that the SB treatment has an 80% chance of being best, our 'Original' has a 17% chance of being the best ...

Google also gives confidence intervals for the conversion rates for each treatment, with boxplots and (95%) credible interval statistics:

The grey bar for the baseline is mirrored in all rows. The 95% CI for the 'improvement over the baseline' is given on the right. But this is a rather wide interval. More informatively, if we hover over the image, we are given more useful breakdowns:

Although this does not exactly tell us the 50% interval 'improvement over the baseline' (this would need a separate computation), we can approximately infer this.

But fortunately it is reported in data we can download; see below "Download (top right)".

From that data, we get:

Our 'posterior' probability thus infers (assuming symmetry, I think) that we should put (considering odds ratios, not percentage points)

• a 2.5% chance of SB having an 18% (or more lower rate of conversion than 'Original'

• a 22.5% chance on SB being between 18% worse and 4% better

• a 25% chance of being 4-20% better

We can also combine intervals, to make statements like ...

• a 50% chance of being 4-36% better

• a 50% chance of being 20-76% better

We report on this further, for this particular case, under

There is some repetition (can we 'mirror blocks'?)

Session balance

Above, even though the treatment has been assigned randomly (presumably a close-to-exact 1/3, 1/3, 1/3 split), the number of 'sessions' differs between the treatments ('variants').

Why? As far as I (DR) understand,

• while each individual user (at least if they are on the same machine and browsing with cookies allowed) is given the same treatment variant each time...

• the same users may 'end' a session (by leaving or being inactive for 30+ minutes), and return later, getting the same treatment but tallying another 'session'. This suggests that users in the "Separate Block" (SB) treatment are returning the most (but also see 'entrances' below).

Breakdown over time

The final section gives the day to day breakdown of the performance of each treatment, presumably, along with confidence intervals. This seems relevant for 'learning and improving while doing' but possibly less relevant for our overall comparison of the pages/treatments.

Download (top right)

The 'Analytics data' gives us sessions and conversions by day and by treatment.

(Where no session occurs in a day for a treatment, it is coded as blank).

Clicking on 'view in analytics'

... this gives some other information, mainly having to do with the user experience.

"Unique page views" represent "the number of sessions during which that page was viewed one or more times." ... Recall "sessions" are periods of continuous activity.

"Entrances" seem potentially very important. According to Google:

Sessions are incremented with the first hit of a session, whereas entrances are incremented with the first pageview hit of a session.

In the present context, this suggests that the 'Separate block' page is inspiring users to come back more often, and to spend more time on average.

Sessions vs. Users

As noted, essentially: 'Sessions' are the number of 'continuously active' periods of an individual user

Analytics measures both sessions and users in your account. Sessions represent the number of individual sessions initiated by all the users to your site. If a user is inactive on your site for 30 minutes or more, any future activity is attributed to a new session. Users that leave your site and return within 30 minutes are counted as part of the original session.

The initial session by a user during any given date range is considered to be an additional session and an additional user. Any future sessions from the same user during the selected time period are counted as additional sessions, but not as additional users.

See Geographic segmentation/blocked randomization for a mainly theoretical discussion of this

Facebook split-testing issues for how to do split testing on Facebook, and the limits to traditional design given their setup