Say you’re running an A/B test: both sides have 10 users, every user converts.
What if one of those users is counted twice? What if one user sees both A and B, converts on A, but gets counted towards both sides?
- 10 conversions / 10 users who saw A = 100% conversion rate
- 10 conversions / 11 users who saw B = 91% conversion rate
The true conversion rate is 100% for both sides, but when we double-count a user suddenly there’s a 9-point gap.