> For the complete documentation index, see [llms.txt](https://globalimpact.gitbook.io/untitled/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://globalimpact.gitbook.io/untitled/partners-contexts-trials/one-for-the-world-oftw/pre-giving-tuesday-email-split-test-+.md). # Pre-giving-tues. email A/B **Context**: Donation 'upsell' to existing pledgers **Question: Are effectiveness-minded (EA-adjacent) donors and pledgers more motivated to donate by** 1. "A": (non-quantitative) **presentation of impact and effectiveness** (as in standard OftW pitch) 2. "B": **Emotional appeals and 'identified victim' images** {% hint style="info" %} **Further information on experiment and outcomes in** in-depth replicable analysis, organized in dynamic document [here](https://daaronr.github.io/eamt_data_analysis/chapters/oftw_upsell_input_first_analysis.html) {% endhint %} ## General idea, main 'hypothesis' **Are effectiveness-minded (EA-adjacent) donors and pledgers more motivated to donate by** 1. "A": (non-quantitative) **presentation of impact and effectiveness** (as in standard OftW pitch) 2. "B": **Emotional appeals and 'identified victim' images** *In the context of **One for The World**'s (OFTW) 'giving season upselling campaign', potentially generalizable to other contexts.* ***Academic framing***: "Does the Identifiable Victims Effect (see e.g., the meta-analysis by Lee and Feeley, 2016) also motivate the most analytical and committed donors?" ## Background and context **One for The World**'s (OFTW) 'giving season upselling campaign'' > 10 emails total over the course of November were sent in preparation for GivingTuesday ### Point of contact (at organization running trial) [Chloë Cudaback](mailto:undefined)\ \ Academic-linked authors: David Reinstein, Josh Lewis, and potentially others ### Timing of trial [Targeted dates](#user-content-fn-1)[^1]: November 10, 18, 23, all in 2021, but may be delayed for feasibility ### Digital location where project 'lives' (planning, material, data) Present Gitbook, Google doc linked below, preregistration (OSF), and github/git repo ### Environment/context for trial Emails\ ... to existing OftW pledgers (asking for additional donations in Giving Season) > All 10 emails had the same CTA: make an additional $100 donation for the giving season/GivingTuesday on top of their recurring monthly pledge donation. ### **Participant universe and sample size** Roughly 4000 participants, as described. A series of three campaign emails will be sent out by OftW to their regular email lists, to roughly 4000 participants, as described. ### Key treatment(s) Basically[^2]: * A list of \~4500 contacts (activated pledgers) was split into two treatment groups. * Treatment Group A received emails that were focused on the contact's impact * while Treatment Group B received emails that were focused on individual stories of beneficiaries See [preregistration, treatment specifics](https://effective-giving-marketing.gitbook.io/untitled/partner-organizations-and-trials/one-for-the-world-oftw/pre-giving-tuesday-email-split-test-+/preregistration-oftw-pre-gt#treatment-specifics-i.e.-experimental-conditions) ### Treatment assignment procedure See preregistration [How many ... conditions](https://effective-giving-marketing.gitbook.io/untitled/partner-organizations-and-trials/one-for-the-world-oftw/pre-giving-tuesday-email-split-test-+/preregistration-oftw-pre-gt#4-how-many-and-which-conditions-will-participants-be-assigned-to) ### **Outcome data** Targeting: Donation incidence and amount in the relevant 'giving season' and over the next year, specifically described in prereg under [key dependent variable](https://effective-giving-marketing.gitbook.io/untitled/partner-organizations-and-trials/one-for-the-world-oftw/pre-giving-tuesday-email-split-test-+/preregistration-oftw-pre-gt#3-describe-the-key-dependent-variable-s-specifying-how-they-will-be-measured.)
**Data storage/form:** * MailChimp data (Chloe is sharing this), * Reports on donations (Kennan is gathering this) ### **Optional/suggested additions** *Planned analysis methods, preregistration link* [here](https://effective-giving-marketing.gitbook.io/untitled/partner-organizations-and-trials/one-for-the-world-oftw/pre-giving-tuesday-email-split-test-+/preregistration-oftw-pre-gt) *Cost of running trial/promotion: Time costs only (as far as I know)* ## Proposed/implementing design (language) ([Link)](https://docs.google.com/document/d/1VyAtfJ2bFaQBfQVlflIdsN29Otr7g8YjjihXVfBv7UM/edit?usp=sharing) {% embed url="" %} ## Pre-registration work Pre-registered on OSF in 'AsPredicted' format, content incorporated here [here](https://effective-giving-marketing.gitbook.io/untitled/partner-organizations-and-trials/one-for-the-world-oftw/pre-giving-tuesday-email-split-test-+/preregistration-oftw-pre-gt) {% embed url="" %} ## Preliminary results ### **Overview:** The Emotion treatment leads to significantly *fewer* people opening emails, but more people clicking on the in-email donation link (relative to the standard Impact information treatment). However, we are statistically underpowered to detect a difference in actual donations. More evidence is needed. Chloe: **those emails that appealed to emotional storytelling performed better (higher in-email click rate) than those that were impact-focused**. **DR, update:** I confirm that this is indeed the case, and this is statistically significant in further analysis. > **Evidence on donations** *(preliminary; we are awaiting further donations in the giving season) ...* {% hint style="info" %} *This is 'hard-coded' below. I intend to replace this with a link or embed of a dynamic document (Rmarkdown).*\ \ \&#xNAN;***The quantitative analysis itself, stripped of any context and connection to OftW, is hosted*** [***HERE***](https://rethinkpriorities.github.io/methodology-statistics-design/inference-and-rough-equivalence-testing-with-binomial-outcomes.html#how-likely-are-proportions-this-similar-under-different-size-true-effect-sizes) {% endhint %} {% embed url="" %} *the analysis as a 'methodological example'; all context removed* {% endembed %} *** {% hint style="info" %} **Note:** We may wish to treat the 'email send' as the denominator, as the differing subject seemed to have led to a different number of opens {% endhint %} *** **Treatment 1 (Impact)**: We record * 1405 unique emails listed as opening a ‘control’ treatment email * 29 members clicking on the donation link in an email at least once (2.1% of openers) * 15 members making some one-time donation in this period (about 0.11% of openers, 0.075% of total) * 8 members emails donating (likely) through the link (0.057%/0.04%) **Treatment 2 (Emotional storytelling):** * 1190 unique emails listed as opening an email (a significantly lower 'open rate', assuming the same shares of members were sent each set of treatment email) * 56 members clicking on the donation link in an email at least once (4.7% of openers) * 11 members making some one-time donation in this period (about 0.9% of openers, about 0.055% of total) * 9 unique emails donating (likely) through the link (0.08%/0.045%) {% hint style="info" %} **Note:** We may wish to treat the 'email send' as the denominator, as the differing subject seemed to have led to a different number of opens {% endhint %} *** **‘Initial impressions of preliminary outcomes’** * The conversion rates are rather low (0.5%) … but maybe high enough to justify sending these emails? I’m not sure. * While people are more likely to O\_pen\_ at least one Impact email, they are more likely to *Click to donate* at least once if assigned the Emotion email * But we can't say much for *actual donations.* * Given the low conversion rates we don’t have too much power to rule out ‘proportionally large’ differences in conversion rates (or average amounts raised) between treatments … ![](/files/JcKMaxWrsipxogZ1o9yk) The figure above seems like a good summary of the ‘results so far’ on ‘what we can infer about relative incidence rates’, presuming I understand the situation correctly …I plot\ \ Y-axis: ’how likely would a difference in donations ‘as small or smaller in magnitude’” than we see in the data between the incidence … against\ \ X-axis: if the “true difference in incidence rates” were of these magnitudes Implementation and management: Chloe Cudaback, Jack Lewars * Our data is consistent with ‘no difference’ (of course) … but it's also consistent with ‘a fairly large difference in incidence’ * E.g., even if one treatment truly lead to ‘twice as many donations as the other’, we still have a 33% chance or so of seeing a difference as small as the one we see * We can reasonably ‘rule out’ differences of maybe 2.5x or greater * Main point: given the rareness of donations in this context, our sample size doesn’t let us make very strong conclusions in either direction about donations [^1]: Update -- four email dates are recorded in the data [^2]: This is Chloe's description... --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://globalimpact.gitbook.io/untitled/partners-contexts-trials/one-for-the-world-oftw/pre-giving-tuesday-email-split-test-+.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.