# Simple quant design issues

## Resources

*Todo: Integrate further easy tools and guides, including those from Jamie Elsey*

{% embed url="<https://daaronr.github.io/metrics_discussion/power.html>" %}

## "Even a few observations can be informative"

Drawing from Lakens' excellent resource:

{% embed url="<https://lakens.github.io/statistical_inferences/bayes.html>" %}

![](/files/Vz3bZBNLrjiHDJya7X5Y)

*You are considering a new and an old message.*

*Suppose you are a ‘believer’* … your prior (light grey up to) is that ‘this new message nearly always performs better than the control treatment’

Suppose you observe only 20 cases and the treatment performs better only half the time. You move to the top black line posterior. You put very little probability on the new message performing much better than the control.

Now suppose you have the ‘Baby prior’, and think all of the following ten things are equally likely

* less than 10% of people rate the new message better than the control
* 10-20% of people rate the new message better than the control
* …
* … 50-60% of people rate the new message better than the control
* ...
* 90-100% of people rate the new message better than the control

You run tests on 20 people, and you get 15 people preferring the new message.

![](/files/Z2rqqgfdnsq0Fr2Sabys)

Now you update substantially. From some calculations (starting from Lakens' code, `pbeta(0.65, aposterior, bposterior)`) you put about an 80% posterior probability that the new message is preferred by at least 65% of the population. (And only about 1.5% probability on the control being better)

So if I really ‘am as uncertain as described in the example above’ about which of two messages are better (and by how much)...

... then even 20 randomly-selected people assessing both messages can be very informative. How often does this ‘strong information gain’ happen? Well, under the "baby prior", you would get information at least this informative in one direction or the other about half the time.<br>


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://globalimpact.gitbook.io/untitled/methodological-discussion/simple-quant-design.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.