A/B Testing « Datadial Blog
0208 6000 500

On the subject of A/B Testing


Tad Chef

May 21st, 2015.

Formulating the Value Proposition so that People Convert Right on the Homepage



There is a lot of advice on how to reduce bounce rates, optimize landing pages and create calls to action. Yet

the most important aspect of converting websites, the value proposition gets seldom mentioned.

Ideally already your slogan contains it and your potential users are not forced to watch a long video or read a wordy text. Also sending your visitors down a link or two makes most of them disappear. How to convert people right on the homepage?


Value Proposition? Are you kidding me?


First off, what the heck is a value proposition? My products represent the value, I don’t have to propose to anybody! Keep calm and read on. The value is obvious to you because you deal with your products or services on a daily basis.

Outsiders, potential clients or online supporters may not even understand what you are talking about. In the worst case they might assume your whole industry is just a bunch of crooks and liars. That’s certainly a problem for us when it comes to SEO and wide-spread prejudice.

The value proposition is not a promotional aspect of your advertising.

It’s not a marketing gimmick. It’s the essence of your business. It’s the actual offer. It’s your unique selling point. Consider a brand like Rolls Royce for example. What’s their value proposition? A car? Probably not.

The car is the product but the offer is the luxury and the symbolism. After all kings and queens ride Rolls Royce limousines. Unlike a renowned brand most average businesses do have to explain their value proposition n their site right away or their visitors might disappear in an instant.


Your homepage is like a hotel lobby



Imagine that you run a hotel. You want to convert your visitor when s/he enters right away in the lobby. Make the lobby minimalist or lavish depending on what your business is about but don’t plan to show everybody around the whole building and suggest them dozens of rooms. Make sure the facade and the neon signs outside are already appropriate.

When someone enters make sure that the temperature is pleasant. I could on like that for a while but we have to return the more abstract realm of websites. At least here we don’t need heating or air conditioning. I hope you understand by now that

a potential guest does not need to inspect all of the rooms to stay with you.

In short the first impression counts. Especially as you assume that the people who arrive in your lobby are already likely to look for a room. They may look for a restaurant as well or a spa but most of them want a place to stay for at least a night.


You don’t have to explain what a hotel is to most of them. This may be different in case your business model is a bit less evident. In any case it’s important to clarify why they should stay with you not the other hotel around the corner or down the street.

I have worked for a few hotels in the past – that is I optimized their sites – so that I know how to differentiate. One of them was a design hotel with a very good connection to to main train station in town. You could actually walk over there. We stressed the two aspects. What is your unique asset others can’t copy easily?

  • What’s the unique flavor?
  • What’s the local specialty?
  • What’s the “killer” feature?

Try to fit that into a short sentence and put in big letters on your frontpage. Yeah, I know. It’s difficult. You’ll probably will struggle to word it in a concise manner. The outcome may sound more like a short paragraph. Then you need to cut all the unnecessary words mercilessly or at least try to prioritize certain parts by using different font sizes:



Testing your value proposition

You are probably familiar with A/B testing already. In most cases business people test whether a particular headline, button text or design gets more conversions than a previous one. Here again I have seldom witnessed test results or studies dealing with the impact of a changing value proposition.

How can you change your value proposition to get more conversions and thus probably more sales?

Well, just consider Ford. They were one of the first to attempt selling mobility instead of just cars. In an era where everybody owns a smartphone and can take part in car sharing without the need to own a car actually it’s about time to rethink the value proposition of car manufacturers.

Where I live (in Berlin, Germany) a lot of them offer their own branded car sharing brands. The foremost example is probably BMW which also owns the Mini brand by now. Thus you see DriveNow Minis and BMW’s seemingly all over the place around here. Also they not only stand around and block parking space, people use them a lot.

By changing the value proposition from something like “owning a cool car with a modern image” to “driving a car right now or whenever you want” BMW was able to adapt to the current market. It may be also possible in your area of expertise. Try different value propositions to find out how your company can adapt to the current trends.

* (CC BY-SA 2.0) Creative Commons image by That Hartford Guy

** (CC BY 2.0) Creative Commons image by Jun Seita



Martina Martina

August 23rd, 2013.

Putting an end to creepy emails!


Spend enough time wading through spam emails and you’ll be amazed at what you find. From dodgy salutations to cringe-worthy formatting, on a very slow day this makes for a good few minutes of entertainment.

Here are some of the most ‘creepy’ elements of bad email-ship:

Over personalisation

Why it’s done & why it sucks:

This is usually an attempt at making a company look uber-friendly, since the emails they send you are formatted like a buddy would send them.

The downside is that sometimes we sign up with nick-names, tags or misspellings which can quickly turn a harmless greeting into a spammy annoyance. Seeing “Happy Birthday JaneDoe101!” or “JaneDoe101 we miss you!” littering up your mailbox is usually the first step on the way to an unsubscribe!

Archaic greetings

Why it’s done & why it sucks:

Dear Sir/Madam” or “To whom this may concern” are clear-cut indicators of cold calling (or cold mailing) – sure they’re gender neutral but boy are they impersonal. They scream “Someone, anyone – please read me!” rather than seeming relevant to the recipient. 

Too many imperatives

Why it’s done & why it sucks:

It’s true that the call-to action is a huge part of the science behind a smooth conversion, so putting in punchy phrases like “click here!” and “buy now!” seem standard procedure, but there is such a thing of over-doing it.

Ultimately, I don’t want bossy emails, and legally anything being sold is an ‘invitation to treat’ so in your next sales email, try the passive approach with an A/B test to see what gains a better reaction.

Too long

Why it’s done & why it sucks:

It’s been a while and there’s a lot to say, but how long do you think I’ve got? People are time conscious, say less!

If you need to explain something a nifty way of doing so is by linking back to a blog post published on your website. Not only is this considerate of your customers, it’s also potential for indirect conversions; they may just browse other parts of your website…

And in taking my own advice, I’ll keep this short and sweet – but feel free to add any other examples you can think of in our vowel to make electronic mail, sustainable! ;-)


August 26th, 2011.

How a losing variation produced 95.45% increase in conversion rate

A/B testing can lead to impressive gains and surprising results. In this post I would like to talk about our recent home page A/B test on an ecommerce store, when our treatment page decreased the goal we were measuring in Google Website Optimizer (GWO) by -11.2%, but still we declared the page a winner. And what a successful winner that was!

How could this be possible?
First of all, when we set up our test on the client home page, in GWO we set up as a success goal ‘free samples requested’ confirmation page. This was done at our client’s request, who believed that this was the most important success metric on his site, forgetting another metric like sales conversion rate.

After running the test for 2 weeks and calculating statistical validity of other site metrics using a Chi-Square statistical validation methodology, we concluded that we collected enough data to establish the treatment page, as a winning page, even though we decreased conversion rate on the ‘free samples requests’ goal we measured in GWO.

How could this be possible? Firstly, ecommerce stores have several important metrics like page conversion rate, revenue, $ Index value, # of transactions and purchased products etc. So when we analysed these data points we discovered that our treatment page performed as follows:

–    Decreased free samples requests by -11.2%

However, the data from our client Google Analytics tool showed us very interesting results for other important site metrics.

–    Page bounce rate was down by 22.08%
–    $ index value up by 65.83%
–    Page conversion rate up by 95.45%
-    Revenue up 46.05%
–    Number of transactions up by 65%
–    Number of purchased products up by 46.42%


Original (control)


Variation 1 (treatment)

One interesting improvement we also observed was that our treatment page sent by over 300% less  visitors to the ‘free samples request’ page, but of those visitors who we sent there, instead of 5.92% we converted 25.11%, thus increasing conversion rate by 324.15%. To remind you, we didn’t make a single change on that ‘free samples request’ page. This was achieved by adding more clarity to the offer on the test page, where we clearly stated what you get after you click.

As you can see, if we simply measured just one goal, then our treatment version would be declared as a loser, even though the treatment page increased monthly gross revenue by over 46%.



When running any test, don’t forget that there are other important metrics which could be impacted by your test and that in some cases, the goal you might think is important isn’t necessarily always the best metric.
Always use your analytics tool, and look at other metrics too, as you don’t want to unknowingly declare a test as a losing one for failing in a certain metric when in fact that change brought extra £20k in revenue and improved another 5 site metrics – as happened in this example.



Interested to find out how Datadial can help you increase your conversion and site revenue? Call us on 0208 6000 500, or request free conversion rate optimisation consultation & site analysis.



May 3rd, 2011.

Home page A/B Test – conversion rate increased by 24.8%

Goal of the test – increase number of downloads
Technology used – Google Website Optimizer
Test duration – 5 weeks
Confidence level achieved – 98.6%

We conducted an A/B test on Dowce.com website to find out which version would increase number of downloads. Both versions were similar, but in the version B (treatment) we improved the following:

  • removed the call to action from the header in order to focus visitors on our main call to action in the body
  • improved the headline
  • added bullets points with improved value proposition
  • improved the call to action (bigger and more visible)
  • improved the layout, by dividing the sections so it is easier to scan the page

A: Control                                                             B: Treatment – 24.8% improvement

This A/B test showed that conversion rate optimisation is very effective marketing method with measurable results, which allows you to increase your revenue and conversion from your current traffic.

To see, what difference a change in conversion rate by 20% or more can make to your business download our conversion improvement calculator.

If you are looking to improve your website conversion rate or would like to learn how we can help request your free consultation at info@datadial.net or call 0208 6000 500.


April 19th, 2011.

Quick test idea to improve your conversion rate – site search result page test. Part 2

In the site search box test part 1, we discovered by analysing web analytics data that visitors using site search are 2-3 times more valuable than those that don’t use site search. Conversion rate difference was also by 168% higher when site search was used. Also, per visit value was 132% greater when site search was used.

In the second part I would like to talk about more advanced, more costly but probably more rewarding test – a site search result page test.

Who is this test for?
Any website where the site search result page doesn’t allow result filtering by category, price, brand etc… or simply looking to improve search functionality, stickiness, conversions and retention by using more robust, intelligent 3rd party site search.

Goal of the test
We would like to find out if using more sophisticated 3rd party ecommerce site search will help you to increase your conversion rate and whether the investment is worth the money.

Why should you test?
You read a case study that better search result pages convert more and that visitors who use site search convert 2 – 3 times better than those not using a site search. You get very excited and decide to approach a 3rd party search provider. However, you discover the investment into a new site search can be substantial (e.g. $25,000+ annually for Google ecommerce search) so you would like find out if you can justify the cost.

How should you test?
What you want to do is to test both site search engines at the same time, something like A/B test comparing 3rd party site search vs. yours. However, there is a caveat. A/B testing won’t work in this case, as the site search box, where it all starts, is on every single page on your site and you would like to capture any search query from any page on your site. So, for this reason we need to set up a site wide test using a multivariate test strategy, showing exactly the same site search box, but the results leading either to your current site search or to the 3rd party site search you are trying to evaluate.

How do you measure success?
By comparing number of sales and a conversion rate in your testing tool

How many visits with site search do I need?
At least 5,000-10,000 visits per month with site search. For fast and statistically valid result 20,000+.

Recommended solutions:
Nextopia – $950/year for 10,000 search queries, $2,495/year for 100,000 search queries
Sli-systems – price not known
Google – from $25,000/year

Contact Jan to see how we can help you with your  ecommerce site search testing, A/B or multivariate testing or to request your free consultation: jan@datadial.net or call 0208 6000 500 ext. 231


April 18th, 2011.

Quick test idea to improve your conversion rate – site search box test. Part 1

Even with all the buzz about conversion rate optimisation, there are still businesses which haven’t tried it yet. To get you started, I came up with a very simple and inexpensive idea every ecommerce store should test. Your site search box position & size test.

OK, so what’s so special about the site search and why should you give it a go?

1. Generally the conversion rate of visitors who used a site search is 2-3 times higher than those without, as is the per visit value. In our example below conversion rate difference is 168% higher when site search has been used. Also per visit value is 132% greater.

2. On average, around 8-20% of sites’ visitors use a site search, but they bring between 17-33% of overall revenue.

3. The test itself isn’t that difficult to implement and could be up and running in less than 8 hours, however this type of test and implementation will require a knowledgeable person to set up the test.

A little something about the site search box test: this type of tests falls under so called “site wide” tests. With site wide test implementation, the tested variations are going to be the same on every page on your website.

Speak to Jan for more information on A/B or multivariate testing or to request your free consultation: jan@datadial.net or call 0208 6000 500 ext. 231

Recent Posts »

Our work »

What we do »

Who we work with »

Got Questions? Lets Talk »