Facebook split-testing issues
Facebook trials: "divergent delivery" --> limited inference
The main point
Facebook serves each ad variation to the people it thinks are most likely to click on it.
Thus, in comparing one ad variation to another... you may learn:
"Which variation performs best on the 'best audience for that variation' (according to Facebook)"
But you don't learn "which variation performs better than others on any single comparable audience."
Update 4 Oct 2022: We may have found a partial solution to this, with ads targeting 'Reach' rather than optimizing for other measures like 'clicks'. We are discussing this further and will report back.
Researchers are interested in running trials using Facebook ads. However, inference can be difficult. Facebook doesn't give you full control of who sees what version of an advertisement.
With A/B split testing etc: They have their own algorithm, which presumably uses something like Thomson sampling to optimize for an outcome (clicks, or a targeted action on the linked site with a 'pixel'). Statistical inference is challenging with adaptive designs and reinforcement learning mechanisms. As the procedure is not transparent, it is even more difficult to make statistical inferences about how one treatment performed relative to another.
Segmentation and composition of population: Facebook's 'PageRank' algorithm determines who sees an ad. I don't think you can turn this off.
We haven't found a way to be able to set it to "show all versions of an ad to comparable populations"
(And even if you could, it would be difficult for you to specifically describe "which population" your results pertain to.)
Divergent delivery and "the A/B test deception"
Last updated