Unfortunately, testing often degenerates into a trial. New designs are somehow thrown into a variant test. The testers hope to win the jackpot. It’s like digging gold somewhere along some river. This is very time-consuming and not really profitable in the long run – unless you are very lucky.

It’s better to keep a short and concise record of what you want to change, why, where and how. How to test and how to measure it. Attached a fact sheet for split testing as we have already used it.

Such fact sheets are framing the context of the test process and it focuses people’s attention who are involved in creating and setting up the test. Even before the detailed design of the test concept, one should be aware of what one wants to test in concrete terms. The first step is to look at the test idea and the underlying hypothesis for a test.

The fact sheets are also a handy tool for the development of a long-term test strategy and corresponding test series. For example, if you want to test the entire ordering processes in several stages. This requires a clear goal, steps that build on each other in a meaningful way – and sometimes also endurance (weeks and months). With the help of the fact sheets, you can tell the “test story” and keep a better overview.

Below the article, you’ll find a link to a small template, with which you can quickly record your first test ideas and hypotheses. But before that, I’d like to make a few comments.

The Hypothesis

Many tests lack a (good) hypothesis, which is crucial to the outcomes. A good hypothesis does not guarantee a successful test. However, it guarantees at least a gain in knowledge.

A thoroughly researched hypothesis doesn’t guarantee a winning test. What it does guarantee is a learning opportunity, no matter the outcome (winner, loser, or inconclusive experiment.) – How to Write a Solid A/B Test Hypothesis

Formulating a crisp test hypothesis is actually not that difficult and always follows a similar structure.

We observe that _______

If we _______ users can/will ________

Thereby ________

If one’s already failing in formulating a  reasonable hypothesis, it’s not worth a test. A hypothesis is not: we make the button bigger and therefore the conversion rate increases…

Let’s get a little more specific with the following example.

Problem

From the hypothesis… We observe that

No test into the blue. If we’re gonna do a test, it’s not for the test’s sake. It’s because we want to solve a certain problem. So we need a description of the problem. Maybe like this: after having re-designed the product page, the CTR (Click Through Rate) to the order process has decreased. Maybe we changed the arrangement of elements on the page, changed wordings or whatever, and for the users, it’s not obvious where to step into the order process.

Solution

From the hypothesis… If we

Describes the adjustments/changes that are made to solve the described problem or achieve the goal. If we’d change the button wording from “continue” to “get started”, this will solve the problem. Sure, that’s a really trivial example 😉

Impact

From the hypothesis… users can/will

Example: More users will then click on the button and go a step further in the ordering process.

From the hypothesis… Thereby

This means that – same user behavior implied – more users will go through the ordering process.

Goals and KPIs (Key Figures) – Measuring success

These are mostly conversion targets, by means of users will perform (or won’t) certain actions, which should have a positive effect on registrations, conversion rates, sales, etc. in general. Their achievement can be measured by means of measurable key figures (KPIs).

In the example above, the conversion rate and thus the orders should increase. But it’s not only the overall result (macro conversion) that can be measured. But also the CTR or Click Through Rate (Micro Conversion) of the page on which the change was made. Most of all affected by the clicks on the button (Micro Conversion), which are also measurable. In this example, we would at least have the following key figures:

  • CR (Conversion Rate) – Macro Conversion
  • Orders– Macro Conversion
  • Clicks on the button – Micro Conversion
  • CTR – Micro Conversion

If we do not reach our goal – to lead more users in and through the ordering process – then we will have good starting points for further steps and tests based on the various measuring points.

Get the Fact Sheet Template

I saved the template as a Google spreadsheet: Fact Sheet

Further Reading

Last but not least, a small collection of resources for in-depth reading about split testing and building hypotheses: