I've started to dive into Facebook advertising and other areas where you are making critical decisions based on the conversion rate of sales pages. As I've gotten into this, I've become more and more concerned about how accurate that number actually is, and what it really means.
The issue is that the conversion rate tells you what just happened, but not how likely it is to happen in the future. It's ability to determine the future is largely dependent on how many people visited the page. And this is something that makes intuitive sense to us.
Imagine we decide to play a game where we flip a coin and I get a dollar for every heads and you get a dollar every tails. If after 4 flips it has come up heads 3 times, you probably aren't going to be worried. But if after 1,000 flips it has come up heads 750 times, you're going to call me a cheater!
The same thing is true with conversion rates. Most of us would agree that if 10 people came to your sales page, and 4 of them bought, it's a little hard to say that in the future 40% of people will also buy. And if a million people came, and 40% bought, we'd feel very comfortable basing predictions on it.
So my question was how do you get a good idea of what the actual conversion percent is, without needing a million visitors. After a bunch of research, I came across the Wald Confidence Interval, which is a pretty neat metric. At a high level, it tells you how many conversions you can actually expect if you had the same amount of visitors again.