How a losing variation produced 95.45% increase in conversion rate
On August 26th, 2011 Jan wrote on the subject of A/B Testing.
A/B testing can lead to impressive gains and surprising results. In this post I would like to talk about our recent home page A/B test on an ecommerce store, when our treatment page decreased the goal we were measuring in Google Website Optimizer (GWO) by -11.2%, but still we declared the page a winner. And what a successful winner that was!

How could this be possible?
First of all, when we set up our test on the client home page, in GWO we set up as a success goal ‘free samples requested’ confirmation page. This was done at our client’s request, who believed that this was the most important success metric on his site, forgetting another metric like sales conversion rate.
After running the test for 2 weeks and calculating statistical validity of other site metrics using a Chi-Square statistical validation methodology, we concluded that we collected enough data to establish the treatment page, as a winning page, even though we decreased conversion rate on the ‘free samples requests’ goal we measured in GWO.
How could this be possible? Firstly, ecommerce stores have several important metrics like page conversion rate, revenue, $ Index value, # of transactions and purchased products etc. So when we analysed these data points we discovered that our treatment page performed as follows:
- Decreased free samples requests by -11.2%
However, the data from our client Google Analytics tool showed us very interesting results for other important site metrics.
- Page bounce rate was down by 22.08%
- $ index value up by 65.83%
- Page conversion rate up by 95.45%
- Revenue up 46.05%
- Number of transactions up by 65%
- Number of purchased products up by 46.42%
Original (control)

Variation 1 (treatment)

One interesting improvement we also observed was that our treatment page sent by over 300% less visitors to the ‘free samples request’ page, but of those visitors who we sent there, instead of 5.92% we converted 25.11%, thus increasing conversion rate by 324.15%. To remind you, we didn’t make a single change on that ‘free samples request’ page. This was achieved by adding more clarity to the offer on the test page, where we clearly stated what you get after you click.
As you can see, if we simply measured just one goal, then our treatment version would be declared as a loser, even though the treatment page increased monthly gross revenue by over 46%.
Conclusion
When running any test, don’t forget that there are other important metrics which could be impacted by your test and that in some cases, the goal you might think is important isn’t necessarily always the best metric.
Always use your analytics tool, and look at other metrics too, as you don’t want to unknowingly declare a test as a losing one for failing in a certain metric when in fact that change brought extra £20k in revenue and improved another 5 site metrics – as happened in this example.
Interested to find out how Datadial can help you increase your conversion and site revenue? Call us on 0208 6000 500, or request free conversion rate optimisation consultation & site analysis.
Why your website isn’t as fast as it should be…
On August 24th, 2011 Martina wrote on the subject of Datadial,Hosting,Inspiration,Internet,SEO,Software,Usability,Web Technology.
Imagine…
…a wheelbarrow in an open field that you drag along every day filling it with this and that – each thing you add to it has some significance and some use.
Now imagine you never empty the wheelbarrow. Each day, not only do the things you found the week before now lie at the bottom covered by the newest additions, but the device also becomes increasingly heavy to pull until eventually, it becomes almost impossible.
Now think of the wheelbarrow as your website, and think of its contents as the factors affecting its speed – Let’s explore these factors…
Bad HTML:
- Empty spaces between code (This only adds to processing time)
- Missing tags (Causing internal errors & bugs in the site)
- Bulky HTML (such as using unnecessary tags where something more CSS compatible would work better e.g. using the tag “font-size” rather than just “small”)
- Background colour being the same as text colour (making all text unreadable)
- Hyperlinks that fail (Devaluing your site in terms of credibility, and possibly increasing bounce rates)
- Missing images
An overload of HTTP requests:
Whenever your web browser fetches a file from a web server, for example when it loads a picture, it does this by using HTTP which stands for “HyperText Transfer Protocol”.
HTTP is an action whereby you’re computer requests for a particular file. One example is a request for ‘home.html‘ (the homepage of a particular website). The web server then sends a response to the computer that says something like: “Here’s the file you asked for” which is followed by the actual file itself.
Understandably, if your server is receiving a very high volume of requests for a range of different things, such as pictures, graphics, photographs, music players and video rendering, it can take its toll and end up really slowing your website down.
JavaScript/Flash overuse:
JavaScript helps make things look nice. Lines of code enable things such as widgets, adverts, and analytics services to work successfully. The issue is that both kinds of software can be “heavyweight”. JavaScript performs ‘sequentially’ rather than ‘concurrently’ – this means that nothing else loads before JavaScript loads. Of course, this becomes an issue when you have tonnes of JavaScript code, each one longer than the last, preventing anything else from happening.
Too many cookies:
HTTP Cookies are used mainly for personalization and authentication purposes. A series of saved information is exchanged between the web server and the browser in order to remember things about how you are using the internet. For example if you are shopping online and exit the website returning at a later date, a cookie will enable the site to remember what you had in your shopping cart so you don’t have to spend time finding the same items again.
However, because saved information is being kept on the server, a build up of this can add to the process time on a website. In some cases, hackers even use cookies as an opportunity to track browsing activity; this is called spyware…so beware!
Bad hosting:
Web hosting is the business of providing storage space and access for websites. Bad web hosting happens when said storage space is overloaded with many websites, yours is added to the list and so runs slow. Other issues caused by a bad web host include:
- Search engines being unable to crawl your site resulting in a fall in Search Rank
- Your website being “down” (not working, sending out 404-errors)
- Not being able to contact your web host to fix the issue (since the service is so bad the system has probably crashed)
Excess of external media:
Embedded YouTube videos, actually embedded anything that is coming from another website can potentially slow yours down. When you embed something from another site, you are relying on that sites web server, that sites speed, and that sites ability to ensure the embedded item is working properly there, so that it works properly on yours site. Often, even when it works just fine, it might add an extra few seconds to a certain page loading…a few seconds a potential customer may be unwilling to wait!
Spam:
Spam is so much more than just a bunch of annoying emails. It slows down the Internet and it increases consumer fees.
The internet is a network where spamming effects everyone that uses it. To push spam around the internet relies on a process; it begins with global networks that pass the spam along to their destination, and ends with the message being received by the recipient.
Simultaneously, time, money and resources are used trying to catch and prevent spammers from infiltrating mail servers resulting in higher costs to the consumer because providers are forced to add more security to their servers and hire more staff to manage and prevent the problem.
Be sure to spam proof all web forms by adding “captchas” or similar.
Favicon neglect:
A ‘favicon’ is an image (as shown above) that stays in the root of your server. It’s definitely needed because even if you don’t care about them, the browser still requests one. If there isn’t one, it will respond with a 404 error (meaning not found). Any error message, such as a 404 or 301, is an extra message sent that adds time to the processing of a site.
This image or lack thereof, interferes with the processing sequence by requesting extra components in the load, and since the favicon is the first thing that is downloaded before these extra components, if there isn’t one, the first thing downloaded will be an error.
Too many advertisements:
Any time a site uses advertisements, you are adding to other processes a site goes through in order to function correctly. Programmes like Google Adsense and Microsoft adcenter are external, and reputable, however it is logical to practice the same rules as with external media; everything in moderation – besides, sites with too many ads look un”site”ly!
If any of these apply to you, take active steps to protect your website against sloth! Speed be with you!
Home page A/B Test – conversion rate increased by 24.8%
On May 3rd, 2011 Jan wrote on the subject of A/B Testing,Company News.
Goal of the test – increase number of downloads
Technology used – Google Website Optimizer
Test duration – 5 weeks
Confidence level achieved – 98.6%
We conducted an A/B test on Dowce.com website to find out which version would increase number of downloads. Both versions were similar, but in the version B (treatment) we improved the following:
- removed the call to action from the header in order to focus visitors on our main call to action in the body
- improved the headline
- added bullets points with improved value proposition
- improved the call to action (bigger and more visible)
- improved the layout, by dividing the sections so it is easier to scan the page
A: Control B: Treatment – 24.8% improvement
Conclusion:
This A/B test showed that conversion rate optimisation is very effective marketing method with measurable results, which allows you to increase your revenue and conversion from your current traffic.
To see, what difference a change in conversion rate by 20% or more can make to your business download our conversion improvement calculator.
If you are looking to improve your website conversion rate or would like to learn how we can help request your free consultation at info@datadial.net or call 0208 6000 500.
Establishing online business credibility with ‘About Us’ page
On April 21st, 2011 Jan wrote on the subject of Design Ideas,Inspiration.
Having a good ‘About Us’ page allows you to show off who you are, what you do and it allows to make people more comfortable doing a business with your company. Unfortunately, this page is being sometimes overlooked, as the perception is that it’s not that important.
However, according to a Stanford Web Credibility Research, they derived 10 guidelines/factors which effects websites credibility, 3 of them relating to your ‘about us’ page (based on a 3 year, 5500 person study):
- Show that there is a real organization behind your site
- Highlight the expertise in your organization
- Show that honest and trustworthy people stand behind your site
Naturally, testing a different version of your ‘About us’ page should be on your to-do test list. Unfortunately, in most cases ‘About us’ pages get at most several hundred visitors per month, which is not enough to come with statistically valid results within your 6 weeks test period. In this case, here is the list of 3 must have things an ‘About us’ page for small and medium businesses should definitely have if you want to get more sales:
3 Must Haves
- Pictures of your office and your team
- A passionate story behind your company showing your values
- Peoples bio (credentials, certificates, training)
Here is the list of some ‘About us’ pages for your inspiration.
Creative examples:
www.technologywithpassion.com (nice animation)
www.tribal.nl (nice animation)
Business (ecommerce + service + corporate) examples:
Quick test idea to improve your conversion rate – site search result page test. Part 2
On April 19th, 2011 Jan wrote on the subject of A/B Testing.
In the site search box test part 1, we discovered by analysing web analytics data that visitors using site search are 2-3 times more valuable than those that don’t use site search. Conversion rate difference was also by 168% higher when site search was used. Also, per visit value was 132% greater when site search was used.
In the second part I would like to talk about more advanced, more costly but probably more rewarding test – a site search result page test.

Who is this test for?
Any website where the site search result page doesn’t allow result filtering by category, price, brand etc… or simply looking to improve search functionality, stickiness, conversions and retention by using more robust, intelligent 3rd party site search.
Goal of the test
We would like to find out if using more sophisticated 3rd party ecommerce site search will help you to increase your conversion rate and whether the investment is worth the money.
Why should you test?
You read a case study that better search result pages convert more and that visitors who use site search convert 2 – 3 times better than those not using a site search. You get very excited and decide to approach a 3rd party search provider. However, you discover the investment into a new site search can be substantial (e.g. $25,000+ annually for Google ecommerce search) so you would like find out if you can justify the cost.
How should you test?
What you want to do is to test both site search engines at the same time, something like A/B test comparing 3rd party site search vs. yours. However, there is a caveat. A/B testing won’t work in this case, as the site search box, where it all starts, is on every single page on your site and you would like to capture any search query from any page on your site. So, for this reason we need to set up a site wide test using a multivariate test strategy, showing exactly the same site search box, but the results leading either to your current site search or to the 3rd party site search you are trying to evaluate.
How do you measure success?
By comparing number of sales and a conversion rate in your testing tool
How many visits with site search do I need?
At least 5,000-10,000 visits per month with site search. For fast and statistically valid result 20,000+.
Recommended solutions:
Nextopia – $950/year for 10,000 search queries, $2,495/year for 100,000 search queries
Sli-systems – price not known
Google – from $25,000/year
Contact Jan to see how we can help you with your ecommerce site search testing, A/B or multivariate testing or to request your free consultation: jan@datadial.net or call 0208 6000 500 ext. 231






















