How a losing variation produced 95.45% increase in conversion rate.

August 26, 2011

A/B testing can lead to impressive gains and surprising results. In this post I would like to talk about our recent home page A/B test on an ecommerce store, when our treatment page decreased the goal we were measuring in Google Website Optimizer (GWO) by -11.2%, but still we declared the page a winner. And what a successful winner that was!

, How a losing variation produced 95.45% increase in conversion rate

How could this be possible?
First of all, when we set up our test on the client home page, in GWO we set up as a success goal ‘free samples requested’ confirmation page. This was done at our client’s request, who believed that this was the most important success metric on his site, forgetting another metric like sales conversion rate.

After running the test for 2 weeks and calculating statistical validity of other site metrics using a Chi-Square statistical validation methodology, we concluded that we collected enough data to establish the treatment page, as a winning page, even though we decreased conversion rate on the ‘free samples requests’ goal we measured in GWO.

How could this be possible? Firstly, ecommerce stores have several important metrics like page conversion rate, revenue, $ Index value, # of transactions and purchased products etc. So when we analysed these data points we discovered that our treatment page performed as follows:

–    Decreased free samples requests by -11.2%

However, the data from our client Google Analytics tool showed us very interesting results for other important site metrics.

–    Page bounce rate was down by 22.08%
–    $ index value up by 65.83%
–    Page conversion rate up by 95.45%
–    Revenue up 46.05%
–    Number of transactions up by 65%
–    Number of purchased products up by 46.42%

Original (control)

, How a losing variation produced 95.45% increase in conversion rate

Variation 1 (treatment)

, How a losing variation produced 95.45% increase in conversion rate
One interesting improvement we also observed was that our treatment page sent by over 300% less  visitors to the ‘free samples request’ page, but of those visitors who we sent there, instead of 5.92% we converted 25.11%, thus increasing conversion rate by 324.15%. To remind you, we didn’t make a single change on that ‘free samples request’ page. This was achieved by adding more clarity to the offer on the test page, where we clearly stated what you get after you click.

As you can see, if we simply measured just one goal, then our treatment version would be declared as a loser, even though the treatment page increased monthly gross revenue by over 46%.

Conclusion

When running any test, don’t forget that there are other important metrics which could be impacted by your test and that in some cases, the goal you might think is important isn’t necessarily always the best metric.
Always use your analytics tool, and look at other metrics too, as you don’t want to unknowingly declare a test as a losing one for failing in a certain metric when in fact that change brought extra £20k in revenue and improved another 5 site metrics – as happened in this example.

Interested to find out how Datadial can help you increase your conversion and site revenue? Call us on 0208 6000 500, or request free conversion rate optimisation consultation & site analysis.