Test, Test and Test some more: The importance of A/B & Multivariate Testing

  • 0
A/B Testing

Test, Test and Test some more: The importance of A/B & Multivariate Testing

a b testing graph

Some may not like it, or know how to do it, that is why you have Solter Consulting, but Testing is one of the most important ways to know what is working well or not, and how to improve.


“By leveraging the power of experimentation and testing tools, you can run experiments on your site and let your customers tell you what works best.” (Kaushik, 2010, 8)

Quantitative-capable of being measured or expressed in numerical terms.

  1. Comparison: A/B Testing vs. Multivariate Testing (MVT)

A/B Testing is basically using two versions of the same page and assessing which version receives a better reaction by either drawing in more unique visitors or getting these visitors to engage in the desired outcome. The pros of A/B Testing are that it is fairly easy to do and extremely cost effective. It is a great starting point for flushing out the best version of your website pages.

The downside or cons to A/B Testing include a bit of difficulty deciphering exactly what change to the page layout made the difference. No doubt, you may make several changes and there is no way to determine what elements made the most difference. Kaushik cautions us that A/B Testing could mean you are making changes too slowly (Kaushik, 2010, 198). Life on the web is ever changing. It is fast and furious!

Multivariate Testing is a way to test changes to several elements at the same time on one web page (Kaushik, 2010, 198.) This testing is a bit more complicated for the layman. JavaScript tags would be placed on the items being tested and alternatives are uploaded into the testing tool (ex. Google Website Optimizer, Optimost, Sitespect.) The pros of multivariate testing are that it produces dramatic results. It will be clear to you exactly which element created an increase in the desired outcome and what changes to make. A con of MVT is that it is a little more complicated for someone like me! It requires a testing tool, which costs money. Like A/B Testing, this test is one page at a time. Depending on how many pages it takes to get the visitor or customer to the desired outcome, this could be a bit of testing and take a bit of time to get the whole picture and the website fully optimized.

  1. Approach: One of the first things I will want to do on my website is an A/B Test of the top landing page with the highest bounce rate. I will then create an alternative page and see how it fairs. I will be sticking with A/B Testing initially, so it may be worthwhile to test my search, email and comment pages as Kaushik advises. I am trying to get more feedback, engagement and loyalty from visitors, so my subscription page and comment pages may also need good analysis/testing to optimize.

Qualitative-quality or character of something.

  1. Comparison: Usability Studies vs. Online Surveys

Usability Studies help us understand the customer and the why of their actions.

Pros of the usability study are that there are web-based versions that are fairly inexpensive to conduct and can be completed quickly. These tests are golden in that you are face to face with your customer and that is rare for almost any web-based business. Some of the cons are that lab usability tests measure a user’s ability to complete tasks and are a much more involved process and costly. You need a room, subjects and a test administrator. Once you have gathered information, you still need to analyze the data. Kaushik cautions that you will only have a small sampling of the population you serve and thus may skew the data. Ultimately, the effectiveness of this testing lies in eliminating the worst ideas of those presented.

Online surveys-customer/visitor self-administered questionnaires help us to understand the why and give the customer a voice.

There are many pros of the online survey. They are quick, easy and affordable.

The downside or con of surveys is that you will only get a certain type of visitor to answer the survey; therefore, you will not have the complete picture. “Page-level surveys are not good for collecting feedback about intent or complete site experience.” (Kaushik, 2010, 181)

  1. Approach:

I think it would be fun and easy to create a survey and pop it on my website to see what brought visitors to my site. I would like to hear their opinion about what they enjoyed or did not enjoy. I might place a page-level survey after a blog post to see if the content was relevant and of interest, so I can adjust my topics accordingly.

I also like the site-level survey approach to collecting complete data about customer/visitor motivation across the site. Kaushik suggests a pop-up window with 25-28 questions. I, like many, am annoyed by pop-ups but, it is a necessary evil in order to glean quality insights for the future direction of your website.

testing website analytics

Give us a call, we will get you where you need to go! Solter Consulting

Email Subscription

Join the Conversation