Pledge page (options trial)
Last updated
Last updated
The presentation of options on GWWC's 'pledge page' were randomly varied at the individual browser level over a certain period to see which option increased pledges.
A summary of this has been shared as a post on the EA Forum.
Giving What We Can (GWWC) has three giving pledge options, displayed in the 'Original presentation version' below.
From April-July 2021 they ran a trial presenting its 'pledge page' options in three slightly different ways. Considering 'clicks on any button' as the outcome, and a Bayesian 'preponderance of evidence' standard...
"Separate Bullets for Other Pledges" was the most successful presentation. It only showed a box for "The Pledge", with the other options given in less prominent bullet points below. This had about a 20% higher incidence rate than the Original presentation.
"Pledge before Try Giving" was the least successful presentation this was like the one displayed above, but with "Try Giving" in the central position. This had about a 23% lower incidence rate than the Original presentation.
These results may only apply narrowly to the GWWC pledge case, and even here, we have some caveats. However, it loosely suggests that when making a call to action, it may be most effective to present the most well-known and expected option most prominently, and not to emphasize the range of choices (see further discussion below).
Getting people to take the GWWC pledge may be seen as an important outcome on its own. It on getting people engaged in the Effective Altruism community and other EA activities, such as EA career impact decisions.
GWWC: How can we present pledge options to maximize positive outcomes (pledges, fulfillment)?
General: For those considering making substantial giving pledges (of a share of their income), how does the presentation of these 'pledge options' matter?
Theories and mechanisms to consider:
Tendency to choose 'middle options' (Simonson and Tversky 1992)
Too many options may lead to 'indecision paralysis'
The signaling power of choice; e.g., if there's a 'more virtuous choice' I may feel that my 'middle choice' looks less good by comparison
GWWC has three distinct pledge options, as shown above
1. "Try Giving" (1% of income),
2. "The Pledge" (10% of income)
3. The "Further Pledge" (donate all income above a living allowance).
here (link from October 2020).
Three versions of this page were randomly presented (between 19-21 April and 10 July 2021)
The content of the key 'choice button' part varied between these three versions
"Original:" A block of three (in the order of commitment) 'The Pledge' (10%) in the center and highlighted (see above)
"Pledge before TryGiving": A block of 3 with "Try Giving" (1%) in the center and highlighted
"Separate Bullets for Other Pledges": A single block for 'The Pledge' (10%), with the other pledges given as clickable bullet points below (as well as a bullet for the 'company pledge' ... which had a different presentation in other versions)
The version presented stayed constant according to an individual's IP cookie tracking.
'Everyone going to the above page' within the above time duration.
People interested in GWWC pledges'
Sample size: see below, from Google Analytics
"Original" (Block of 3 in order of commitment, Middle Pledge in Center)
2. "Pledge before TryGiving" ... as above but with Try Giving and The Pledge swapped, and Try Giving (in the center) highlighted
3. "Separate Bullets for Other Pledges" (see below)
Three versions of this page were randomly presented
Equal likelihood of
The version presented stayed constant
Statistics on Google Analytics: This records only 'pressed any button' (any pledge) as the successful outcome.
See Google A/B, optimize interface for details on data extraction from the interface
From shared image from Google Analytics:
'Experiment sessions' (observations) by treatment (as labeled on Google Analytics shared image):
Original: 2588
Pledge before Try Giving: 2686
Separate Bullets for Other Pledges: 2718
Total: 7992 sessions (=2588+2686+2718)
3. Where is the data stored ... [noted above]
The "separate bullets for other pledges" seems to have been the most successful, with an 0.49% higher (percentage point) incidence rate than the 'Original', i.e., a 22% higher rate of pledging (2.69 vs 2.20).
These differences seem unlikely to be statistically significant in a conventional sense. Still, Google analytics (presumably a reasonable Bayesian) model states an 80% chance that this is the best treatment, and this seems useful and informative.
The "Pledge Before Try giving" treatment performed substantially worse than the original.
Downloading the 'Analytics data' behind the above graphs, we see:
Variant | 2.5th Percentile Modeled Improvement | 25th Percentile Modeled Improvement | Modeled Improvement | 75th Percentile Modeled Improvement | 97.5th Percentile Modeled Improvement |
---|---|---|---|---|---|
Original | 0% | 0% | 0% | 0% | 0% |
Pledge Before Try Giving | -50% | -33% | -23% | -11% | 18% |
Separate Bullets For Other Pledges | -18% | 4% | 20% | 36% | 76% |
This suggests it is very reasonable to think that 'Separate Bullets' is substantially better
Our 'posterior' probability thus infers that
a 2.5% chance that 'Separate Bullets' (SB) has an 18% (or more) lower conversion rate than 'Original'
a 22.5% chance on SB being between 18% worse and 4% better
a 25% chance of SB being 4-20% better
a 25% chance of SB being 20-36% better
A 22.5% chance of SB being 36-76% better
A 2.5% chance of SB being more than 76% better
We can also combine intervals, to make statements like ...
a 50% chance of being 4-36% better
a 50% chance of being 20-76% better
For 'Pledge before...' (PB) we can state, e.g.,
PB has a 75% chance of being at least 11% worse than Original
and a 50% chance of being at least 23% worse than Original
Perhaps giving people more options makes them indecisive. They may be particularly reluctant to choose a “relatively ambitious giving pledge” if a less ambitious option is highlighted.
This could also involve issues of self and social signaling. If the 'main thing' to do is a 10% pledge (as in "separate bullets"), then this may seem a straightforward way of conveying 'I am generous'. On the other hand, if the 'Further pledge' is fairly prominent, perhaps the signal feels less positive. And if the '1% pledge' is made central, 10% might seem more than a necessary signal.
The "pledge before try giving" may perform the worst because it makes the 'Try Giving' pledge a particularly salient alternative option. (In contrast, the "Original" at least makes 'The 10% Pledge' the central and the middle option.)
I am assuming that the 'outcome being measured here' is whether the person 'clicked on any giving pledge'; this is what Luke has conveyed to me
I assume this is 'conversions ever from this IP', and 'sessions' represents 'how many different IPs came to the treatment'. If it's something else (e.g., each 'session' is a 'visit' from an individual), this could reflect these people converting in fewer sessions but not necessarily being more likely to convert overall. Even if this is 'by IP' the alternative interpretation 'not converting now but maybe later' may still have some weight if people are entering through multiple devices.