Test, Test and Test some more: The importance of A/B & Multivariate Testing

  • 0
A/B Testing

Test, Test and Test some more: The importance of A/B & Multivariate Testing

a b testing graph

Some may not like it, or know how to do it, that is why you have Solter Consulting, but Testing is one of the most important ways to know what is working well or not, and how to improve.


“By leveraging the power of experimentation and testing tools, you can run experiments on your site and let your customers tell you what works best.” (Kaushik, 2010, 8)

Quantitative-capable of being measured or expressed in numerical terms.

  1. Comparison: A/B Testing vs. Multivariate Testing (MVT)

A/B Testing is basically using two versions of the same page and assessing which version receives a better reaction by either drawing in more unique visitors or getting these visitors to engage in the desired outcome. The pros of A/B Testing are that it is fairly easy to do and extremely cost effective. It is a great starting point for flushing out the best version of your website pages.

The downside or cons to A/B Testing include a bit of difficulty deciphering exactly what change to the page layout made the difference. No doubt, you may make several changes and there is no way to determine what elements made the most difference. Kaushik cautions us that A/B Testing could mean you are making changes too slowly (Kaushik, 2010, 198). Life on the web is ever changing. It is fast and furious!

Multivariate Testing is a way to test changes to several elements at the same time on one web page (Kaushik, 2010, 198.) This testing is a bit more complicated for the layman. JavaScript tags would be placed on the items being tested and alternatives are uploaded into the testing tool (ex. Google Website Optimizer, Optimost, Sitespect.) The pros of multivariate testing are that it produces dramatic results. It will be clear to you exactly which element created an increase in the desired outcome and what changes to make. A con of MVT is that it is a little more complicated for someone like me! It requires a testing tool, which costs money. Like A/B Testing, this test is one page at a time. Depending on how many pages it takes to get the visitor or customer to the desired outcome, this could be a bit of testing and take a bit of time to get the whole picture and the website fully optimized.

  1. Approach: One of the first things I will want to do on my website is an A/B Test of the top landing page with the highest bounce rate. I will then create an alternative page and see how it fairs. I will be sticking with A/B Testing initially, so it may be worthwhile to test my search, email and comment pages as Kaushik advises. I am trying to get more feedback, engagement and loyalty from visitors, so my subscription page and comment pages may also need good analysis/testing to optimize.

Qualitative-quality or character of something.

  1. Comparison: Usability Studies vs. Online Surveys

Usability Studies help us understand the customer and the why of their actions.

Pros of the usability study are that there are web-based versions that are fairly inexpensive to conduct and can be completed quickly. These tests are golden in that you are face to face with your customer and that is rare for almost any web-based business. Some of the cons are that lab usability tests measure a user’s ability to complete tasks and are a much more involved process and costly. You need a room, subjects and a test administrator. Once you have gathered information, you still need to analyze the data. Kaushik cautions that you will only have a small sampling of the population you serve and thus may skew the data. Ultimately, the effectiveness of this testing lies in eliminating the worst ideas of those presented.

Online surveys-customer/visitor self-administered questionnaires help us to understand the why and give the customer a voice.

There are many pros of the online survey. They are quick, easy and affordable.

The downside or con of surveys is that you will only get a certain type of visitor to answer the survey; therefore, you will not have the complete picture. “Page-level surveys are not good for collecting feedback about intent or complete site experience.” (Kaushik, 2010, 181)

  1. Approach:

I think it would be fun and easy to create a survey and pop it on my website to see what brought visitors to my site. I would like to hear their opinion about what they enjoyed or did not enjoy. I might place a page-level survey after a blog post to see if the content was relevant and of interest, so I can adjust my topics accordingly.

I also like the site-level survey approach to collecting complete data about customer/visitor motivation across the site. Kaushik suggests a pop-up window with 25-28 questions. I, like many, am annoyed by pop-ups but, it is a necessary evil in order to glean quality insights for the future direction of your website.

testing website analytics

Give us a call, we will get you where you need to go! Solter Consulting

  • 0
objective. Goal, KPI

Back to Basics-Establishing Objectives, Goals and KPIs

analyticscomponetsSometimes the best thing we can do for our customers is point them in the right direction, help them define their objectives and goals and create a framework for measuring them (KPIs).

Here is a mock up of the building blocks for a robust analytics program.

Section 1: Clickstream-“clickstream is the foundational data which helps you measure pages and campaigns and helps you analyze all kinds of site behavior: Visits, Visitors, Time on Site, Page Views, Bounce Rate, Sources…”(Kaushik 2010, 7)

Section 2: Multiple Outcomes- “Focusing deeply and specifically on measuring Outcomes means connecting consumer behavior to the bottom line of the company” (Kaushik 2010, 8)

Section 3: Testing- “By leveraging the power of experimentation and testing tools, you can run experiments on your site and le your customers tell you what works best” (Kaushik 2010, 8)

Section 4: Competitive Intelligence- “Knowing how you are performing against your competition is priceless – it helps you improve, it helps you identify new opportunities, and it helps you stay relevant” (Kaushik 2010, 10)

Section 5: Reporting- “If we want executives to use our dashboards and take action, then we must give them information and not data” (Kaushik 2010, 288)

Key Metrics:

The key metrics I am interested in are visits to www.xyzwebsite.com and more importantly unique visitors. Bounce rate is important too. I want to know if people hit my site and “puke” as Kaushik states in chapter 3. If my visitors do bounce, what page in particular is causing this reaction, which incorporates exit rate, or a bad exit? Conversion rate is also important to me, as I want to convert unique visitors to subscriptions and even better guest blog submissions. Just as important to the success of www.xyzwebsite.com is Engagement. How did my visitors enjoy their stay? I can measure this in the form of comments, liking and sharing the post on social networks.

Web Analytics Framework:

*Business Objective: What are we trying to accomplish? www.xyzwebsite.com is trying to create brand name recognition through the web.

  1. Get visitors to the site, increase number or visits and unique visitors.
  2. Increase activity through comments and guest submissions, sharing link.
  3. Visitor Loyalty-subscriptions, following on social networks.

The end game is to create profit through click-thru advertising, paid freelance writing assignments, paid reviews and free products/services.

Are these simple objectives DUMB: Doable. Understandable. Manageable. Beneficial?

*Goals: Goals are specific strategies you’ll leverage to accomplish your business objectives. 1. do x 2. improve y 3. reduce z.

  1. Write more blogs-get name out there and search engines working.
  2. Improve SEO
  3. Improve site appearance-fix tags; re-install share buttons on the bottom of every page.
  4. Install Google Analytics program –What are the most popular posts? What time of day are people visiting the site? What is the referring site? Optimize site based on this data.

*Key Performance Indicators: A key performance indicator (KPI) is a metric that helps you understand how you are doing against your objectives.

  1. Number of unique visitors
  2. Number of Subscribers
  3. Number of comments/Guest blog/recipe submissions
  4. Sharing post link-number of likes and shares on social networks

*KPI Targets: Targets are numerical values you have pre-determined as indicators success or failure.

  1. 1000 unique visitors a week.
  2. 50 subscriptions a week.
  3. 2 comments on every post, one submission a week.
  4. 100 likes and 50 shares total.



Solter Consulting can help you get your business headed in the right direction. Contact us today.

Email Subscription

Join the Conversation