The Importance of a / B Design Testing

by on 9th March 2010 with 14 Comments

For many developers, launching a site is not the end of the design process. To continually improve the success of their design, these developers turn to A/B testing. This relatively simple process can teach you loads about what your users are looking for as well as what they ignore or find unimportant.

Today we’ll take a quick look at what A/B testing is, the benefits of implementing it, and some tools to get you going.

What is A/B Testing?

A/B testing is extremely simple in concept. It’s basically testing the effectiveness of different designs to find the optimum solution. These tests are usually performed on live sites with real users who are completely unaware of the test. To clarify, lets look at an example.

Is Bigger Better?

Let’s say Vanilla Forums wanted to know if their current homepage could be improved by increasing the size of the sign up button. The first thing they would do is create an alternate version of their page containing the desired revisions. The image below shows their current page followed by the page they’d like to test:


As you can see, the new version differs enough to theoretically impact the way users perceive the page. However, in order to get a good idea of the impact of the placement of that single button, we haven’t moved much else on the page. You can of course use this method to test the difference between two completely different designs, but the thing I really want to highlight is that A/B testing provides you with solid information on how to improve a current design without making a complete overhaul of the site.

Now Vanilla Forums would setup their site to show version one to half the visitors for a specified amount of time and version two to the rest. Then they would compare the results of the two pages to see which snagged the greatest amount of users (using some tools we’ll discuss later).


I’m sure you can immediately name several of the benefits this method would bring to your own site. Let’s go over a few.

It’s Easy!

First and foremost, moderate A/B testing is neither expensive nor difficult to implement all on your own; no expensive consultants or outside agencies required. It definitely takes some work but fortunately there are some great resources out there to help you do the heavy lifting.

Hard Evidence

Using A/B testing lays to rest any usability arguments you might be having around the office or even within your own mind. If your team is torn over two possible scenarios for your site, you can have the various parties dispute the matter all day long but the bottom line is you simply can’t know for certain what will appeal to your users most without testing it out in the real world. A/B testing provides you with an easy way to prove who is right through actual statistics from user interactions.

Incremental Improvement

I’ll again stress that one of they key pieces of potential here is the ability to incrementally improve the interface and/or messaging on your site. As a site owner, this can provide you with a solid way to improve your earnings over time. As a developer, this could provide you with a way to form long-term contracts with your clients.

Things to Keep in Mind

Despite the straightforward nature of the concept of A/B testing, there are several things you should be aware of and considering as you implement it on your site.

What’s Your Goal?

“Better” is quite the relative term. You can’t know if one page is better than another unless you define the terms of the judgement. Similarly, when performing a test, you can’t run wildly into it without consideration of what it is you want to observe. For instance, the Vanilla Forums example above was meant to increase the number of users that click the button to sign up for the service. Note that this is a clear goal that is easily testable, not simply a vague statement regarding the aesthetic nature of one layout over another.

There are several possible goals and metrics to use when testing the efficacy of one design over another. These include the total time spent on the site, the number of pages visited, the percentage of users who navigated to a specific page, etc.

There Won’t Always Be a Clear Winner

Despite the fact that A/B testing can give you clear proof of the superiority of one design over another, sometimes the result isn’t so clear cut. It’s possible that something that you imagined would play a key role in the actions of the users doesn’t really affect them in any significant way. Further, even if you observe a difference, that doesn’t necessarily mean that the difference was a result of the variable. Basic statistical principles prevent a degree of random fluctuation between the results of each version, the goal is to spot differences beyond what would normally occur even if each of the two groups received the same page. Keep in mind that the more users you have to take part in the test, the greater the accuracy of the results.

Resist the urge to nit-pick every tiny portion of your site and change something every time you see the slightest increase in conversions. Look for significant ways to improve your site through clearly met goals.

Multivariate Testing is Also Possible

If you want to go beyond testing the effect of a single variable, multivariate testing provides a way to change up the mix of elements shown to a given user to track which variables have the greatest impact.

Tools and Resources

Now that we’ve briefly gone over the ins and outs of A/B testing, I’m sure you’re eager to get started. Check out these great resources to get you on your way to enhancing your sites.

Google Website Optimizer


If you’re just getting started with A/B testing, Google Website Optimizer should be your first stop. It’s fairly straightforward and easy to use and has a lot of resources to walk you through the process of testing (A/B or multivariate). Best of all: it’s completely free.

Alternatives to Google

If you’re not impressed with Google’s offerings, check out these other options.

Visual Website Optimizer








More Info

Want to know more about A/B testing? Check out these other great articles.


You should now be equipped to begin testing your site for potential improvements. Get started today and let us know in the comments below what section of your site you think A/B testing is most appropriate for.

Comments & Discussion


  • Amberly | Web Designer

    Very Interesting read.These will come in handy for future projects

  • Pingback: The Importance of A / B Design Testing – Process Overview and Tools « Web Page Authority Blog()

  • Pingback: links for 2010-03-09 | :: Daily Professional Web Design, Development, Programming Freelancer, Hacks, Downloads, Math and being a Web 2.0 Hipster?()

  • Radek

    In my opinion this should be done in a earlier stage i.e. in the design stage. That way you can find even more information. Many times when a user see the final product he/she is stunned by the design or functionality. When keeping it to simple black and white or not finished “sketch” one can get more information/ideas from the end user. They are more critical when they don’t see the final “sketch”/product.

  • Pingback: Saturday Swirl 9: Tasty Snacks From the Design Community | Visual Swirl()

  • Jean-Francois Monfette

    Great article ! I think that A/B and multivirate are starting to catch on. I recently wrote a blog post about it too (in french). I found two great resources about this : Each week, you can vote on an A/B test and see which won was the winner in a real test. Also list 15 A/B and multivirate testing tools with descriptions and user comments. They say they are 100% neutral and commission free.

  • Jean-Francois Monfette

    I meant multivariate, but kept writing multivirate. Sorry for the confusion !

  • Cheap Printing

    I can totally see the reason for this but in reality it’s so hard to do. I have a site which i keep changing but it’s so difficult to see which factors are actually effecting it, i.e time of year, economy or actual design. Guess the only real way to do this is to have 2 domains up with similar sites, content and prices. I’m going to try this, would relaly like to see if the makeup of my sites make any difference to sales.

  • Jordan fans

    I have to say, I enjoy reading your blog. Maybe you could let me know how I can subscribing with it ? I feel I should let you know I found your page through yahoo.

  • slimming products

    I like it very much and which you shared the info in this post is very useful. Thanks for sharing a wonderful post. I found this post while surfing the web.Thanks for sharing this article.

  • Pingback: How to Design a Tour Page: Examples and Best Practices | Design Shack()

  • TA-65

    Our intention can be to evaluation problems identified according even although in the path of distinctive genres of composing.

  • トレンチコート

    Thank you for the information that provided.

  • Geothermal

    Great article. I was researching for a school project. This one helps. Thanks.



About the Author