💡
EA market testing (public)
  • Introduction/overview
    • Introduction & explanation
    • 👋Meet the team
    • 📕Content overview
    • Progress/goals (early 2023)
      • EAMT progress & results
      • Goals, trajectory, FAQs
  • 🤝Partners, contexts, trials
    • Introduction
    • Giving What We Can
      • Pledge page (options trial)
      • Giving guides - Facebook
      • Message Test (Feb 2022)
      • YouTube Remarketing
    • One For the World (OftW)
      • Pre-giving-tues. email A/B
        • Preregistration: OftW pre-GT
    • The Life You Can Save (TLYCS)
      • Advisor signup (Portland)
    • Fundraisers & impact info.
      • ICRC - quick overview
      • CRS/DV: overview
      • 📖Posts and writings
    • University/city groups
    • Workplaces/orgs
    • Other partners
    • Related/relevant projects/orgs
  • 🪧Marketing & testing: opportunities, tools, tips
    • Testing Contexts: Overview
    • Implementing ads, messages, designs
      • Doing and funding ads
      • Video ads/Best-practice guidelines
      • Facebook
      • Targeted ad on FB, with variations: setup
    • Collecting outcome data
      • Facebook ads interface
        • Pivot tables
      • Google analytics interface
      • Google A/B, optimize interface
      • Reconciling FB/GA reports
      • Survey/marketing platforms
    • Trial reporting template
  • 🎨Research Design, methodology
    • Methods: Overview, resources
    • "Qualitative" design issues
    • Real-world assignment & inference
      • Geographic segmentation/blocked randomization
      • Difference in difference/'Time-based methods'
      • Facebook split-testing issues
    • Simple quant design issues
    • Adaptive design/sampling, reinforcement learning
    • 'Observational' studies: issues
    • Analysis: Statistical approaches
  • 🧮Profiling and segmentation project
    • Introduction, scoping work
    • Existing work/data
      • Surveys/Predicting EA interest
      • Awareness: RP, etc.
      • Kagan and Fitz survey
      • Longtermism attitudes/profiling
      • Animal welfare attitudes: profiling/surveying
      • Other data
    • Fehr/SOEP analysis... followup
      • Followup with Thomas Ptashnik
    • Further approaches in progress
      • Profiling 'existing traffic'
  • 📋(In)effective Altruistic choices: Review of theory and evidence
    • Introduction...
    • The challenge: drivers of effective/ineffective giving
      • How little we know...
    • Models, theories, psych. norms
    • Tools and trials: overview
      • Tools/interventions: principles
      • Outcomes: Effective gift/consider impact)
        • (Effectiveness information and its presentation)
        • (Outcome: Pledge, give substantially (& effectively))
          • (Moral duty (of well-off))
        • Give if you win/ conditional pledge
      • Academic Paper Ideas
  • Appendix
    • How this 'gitbook' works
      • Other tech
    • Literature: animal advocacy messaging
    • Charity ratings, rankings, messages
    • "A large-scale online experiment" (participants-aware)
  • Innovationsinfundraising.org
Powered by GitBook
On this page

Was this helpful?

Edit on GitHub
Export as PDF
  1. Research Design, methodology
  2. Real-world assignment & inference

Difference in difference/'Time-based methods'

PreviousGeographic segmentation/blocked randomizationNextFacebook split-testing issues

Last updated 2 years ago

Was this helpful?

Abstract .... While effective, this geo-based regression (GBR) approach is less applicable, or not applicable at all, for situations in which few geographic units are available for testing (e.g. smaller countries, or subregions of larger countries) These situations also include the so- called matched market tests, which may compare the behavior of users in a single control region with the behavior of users in a single test region. To fill this gap, we have developed an analogous time-based regression (TBR) approach for analyzing geo experiments. This methodology predicts the time series of the counterfactual market response, allowing for direct estimation of the cumulative causal effect at the end of the experiment. In this paper we describe this model and evaluate its performance using simulation.

Some specific notes/concerns

Geo experiments” where only a single geo is targeted for a treatment seem fairly common in practice. You ‘try something in a single market 1x only and see what it does’.\

  1. This is probably reinventing the wheel some existing thing in Econometric (difference in difference, event studies?), but what?

  2. I find it strange/suboptimal that they aggregate across the Geos in the control group, throwing important variation here … that might tell us something about how much things ‘typically vary by without treatments’. I wonder if there’s another approach that brings that variation back?

    1. Maybe this is 'because this is an easy extract to get from Google Analytics'? How do we get it?

  3. The is 5 years old with no recent updates … ages in this world; is there something better to use instead

🎨
Estimating Ad Effectiveness using Geo Experiments in a Time-Based Regression Framework Jouni Kerman, Peng Wang, and Jon Vaver Google, Inc. March 2017
DR hypothesis note-taking version
Related software package
package