Better approach to minimum share in roulette #1310
Conversation
| }); | ||
|
|
||
| it('should return second variant', () => { | ||
| it('should return third variant', () => { |
| return { | ||
| variantName, | ||
| weight: Math.max(mean / sumOfMeans, minWeight), | ||
| weight: weight < minWeight ? minWeight : 0, |
There was a problem hiding this comment.
My only concern with this is if you have v1 = 0 and v2 = 0.11. then clearly v2 is performing much better than v1. However, the min weight assignment will increase v1 to 0.1 giving a 0.01 difference to the two versions. This over promotes very poor solutions.
There was a problem hiding this comment.
Fair point, I guess this implementation assumes that variants have a low weight because we don't have enough data, rather than because they're just terrible!
| const weights: { weight: number; variantName: string }[] = minimumWeights | ||
| .map(({ variantName, weight, mean }) => ({ | ||
| variantName, | ||
| weight: weight === 0 ? remainder * (mean / sumOfNonReservedMeans) : weight, |
There was a problem hiding this comment.
If the mean is 0.01 and the sumOfNonReservedMeans is 1, then this would be given a weight of 0.01 -- much lower than a variant at 0 which gets 0.1. Assuming I'm following the code correctly.
There was a problem hiding this comment.
at this point we're setting the weight for any variant that we know would get more than minWeight (0.1), so this should be ok
2 participants
A previous PR ensured that all variants in a "roulette" test get some share of the audience regardless of their score.
This PR refines this logic to ensure they get at least 10% of the share.