In this chapter, we will be discussing on how to measure the results and simplify statistics of the testing done on the website.
Understanding Statistics
Researchers may or may not understand statistics. But, A/B Testing tool proves to be a savior by simplifying those statistics. So a lot of calculation can be avoided. Most of the testing tools are consistent in using a 95% criteria as a successful goal completion.
That means out of 10 you are a winner by 9. Let us take an example. Your testing tool report is as follows −
| Variations | Conversion rate |
|---|
| Control page | 1.91% |
| 1 | 2.39% |
| 2 | 2.16% |
| 3 | 3.10% |
This report predicts a more or less .20% conversion rate variance at the 95% interval. Statistically, the goal range is between 1.76 and 2.06.
Fetching Insights
While planning a test I keep two goals in mind. First, is boosting revenue and the other is fetching insights about what prompted a higher ROI.
For instance, in a case study we get diverting traffic to product page rather than category page or homepage improves conversion rates or not. We took three variations, in one we directed traffic on the homepage loaded with categories and subcategories directing further to the product page. In the second, we directed traffic to category page adding filters. In the third, we directed it directly on the product detail page with buy button.
To my surprise, the third variation won. This is just the information required by a buyer about the product. This lets us learn how conversion rate lift and continuous improvements can make us grow our leads.
Undoubtedly, adding many variations and insights over tests gave us the website redesign.
Understanding Results
Let me clear it out. NOT ALL TESTS WIN. Yes, it is painful yet true.
There are tests that gives you results in flying colors. There are others that after so many tries too would be without a result. But if you plan a test with an insight driving segmentation, you can have a new hypothesis to be tested on. Not all test helps you to improve revenue.
Take an example to understand. There are three campaigns with different conversion rates.
| Campaign A | 8.2% |
| Campaign B | 19.1% |
| Campaign C | 5.2% |
Anyone will blindly say ‘Campaign B’ is a super performer. But let us dig some more.
| Visits | Transactions | Conversion Rate |
|---|
| Campaign A | 1820 | 150 | 8.2% |
| Campaign B | 20 | 4 | 19.1% |
| Campaign C | 780 | 41 | 5.2% |
See closely, ‘Campaign B’ is too small to be statistically significant. Campaign B with 1 transaction with one visit will give 100 percent conversion rate. ‘Campaign A’ performs over ‘Campaign C’. While concluding results, there are several factors that need to be looked at and it may differ every time. It is you who need to look at all insights and decide the results.