'Observational' studies: issues

Example: "Hours spent promoting" vs "number of fellows"

Consider a study where

• EA groups are asked to voluntarily participate (with no direct compensation)

• to report the 'time spent on each recruiting activity',

• and to ask their fellows/members 'how did you hear about our group?'

Suppose this finds

'per hour spent by the organizers, far fewer people report "tabling" as the source, relative to 'a direct email'.

Should we interpret this as

'direct emails are a more efficient use of time than tabling, thus groups should spend less time doing tabling and more time sending emails?'

Maybe, but we should be careful; there are other explanations and interpretations we should delve into. Some of these could be partially addressed through survey design, others through careful analysis. Other 'causality' issues may require an experiment/trial/test to get at.

Statistical inference: chance and selection/selectivity

Random variation: With a small sample of groups, these numbers may be particularly high or low (for tabling, for emails, etc) by chance; the averages for a 'typical group' may turn out to be very different.

• This is the standard issue of statistical inference about a population from a sample.

• The issue of 'misrepresenting the population' tends to be worse with smaller samples (here small number of groups, and small numbers of observed outcomes in each group; e.g., only a few fellows)

• However, 'as Bayesians know' you can still draw valuable decision-relevant inferences from small samples. IMO (Reinstein) the "problem of small samples" tends to be overstated because we mainly learn about statistics designed for a particular scientific frequentist approach.

Selection/selectivity: The groups that 'opted in' to be part of this survey may not be a 'random draw' from the population of relevant groups. It may represent more careful or more enthusiastic groups, perhaps groups that are particularly analytical and not so good socially, etc. If some of the 'fellows' within the groups don't complete the survey, this could add another 'selection bias'.

'Marketing causality' issues

• Attribution with multiple sources: “How did you hear about this program?” This could be interpreted in several ways, probably “how did you first hear”. But in marketing sometimes people hear about something multiple times, and it’s hard to know which of these are pivotal in getting them to take action. (We could probably do something to make this question a bit more informative.)

• “Lift”: some people might have signed up anyways even without the activities identify as ‘how they heard about it’. Other people may have been harder to reach, and for the latter (e.g.) tabling ‘Spoke to us while we were tabling’ may be pivotal.

Decisionmaking implications

• Costs $\neq$ hours: the cost of these activities may not be fully proportional to the times spent … e.g., writing a professor may be mentally costly and possibly cost some other social capital. On the other hand tabling may be fun and social, and also generate interesting feedback (and other benefits that are harder to measure, like links with other groups also doing tabling)
• Diminishing returns/hard limits on some activities … e.g., there may be only so many professors (or students) to email. After a few hours of this