Make the probability of success less than 50% for all topics

[lukasvermeer]

Omer wrote:

Once you identify a good topic you have such a strong prior that the expected uplift on shipping that topic will be positive that you focus only on exploitation with no exploration and limited concern about downside protection or false positives.

Probability of success higher than 50% likely never happens in real life: if we were to ship every idea around a topic our expected impact would likely be negative, it being easier to hurt than improve conversion and due bad ideas, mistakes, bugs etc. So the mean uplift of every topic being negative could be both more realistic and also make the game more balanced between explore vs exploit and force players to pay more attention to confidence levels and run experiments longer. Protecting against false positives and downside would become more important.

I like this idea. I suspect it means effect sizes for each topic would have to be bimodal (one mass on the plus side, one on the negative side) rather than the simple normal they are now.

There would still be value in figuring out topics with heavy mass on the plus side, but shipping that topic blind would result in net negative impact on average.

Another related idea I had was to add “buggy” experiments. So once in a while for any topic an experiment has very negative impact because of a bug. It would have similar effect, I think.