Learn how to run experiments in Unless with split tests, A/B/n tests, control groups, and page goals.
Unless offers multiple ways to test the success of your experiences; split tests, testing multiple variations, continuous validation with a control group, as well as setting page goals.
You can create an experience (on-site or component) and adjust its settings as usual, click save, and move on to the Testing tab. Alternatively, you can use the editor to make changes first and then go to the Testing tab. This way when you are adding more variations, they will also have the changes you made to the first variation.
Once at the testing tab, you will find the following options:
We generally recommend starting with one of the first two options to reach statistical significance faster, after which you can continue with a small control group (10%) for continuous validation or maybe even no control group if you are certain of the benefit of a particular experience.
If you want to "Test multiple variations (A/B/n)" you will need to "Add a variation" or more. Once adding a new variation, a pop-up will appear where you can add a control variation or a new variation with a name. You will then be able to see these variations at the Testing tab.
You can now open up the Editor by choosing one of the variations from the dropdown. Note that the control variation will not show up in the dropdown since it cannot be edited. Once in the editor, you can make your changes to a variation, go back and repeat the steps for the next variation. Remember to save and publish your changes each time!
If you selected any of the other options, you don't need to add a variation, and can directly open up the editor, make changes, save, and publish.
Deleting a variation is also possible but keep in mind that it cannot be undone. After deletion, all visitors will be distributed across the remaining variations. Insights for deleted variations will still be available.
In addition to the testing options mentioned above we also offer the option to set up a page specific goal. Let's say, you made changes to a page and want to see the impact that has on a CTA button getting clicked more or not, you can set up a page goal!
In Unless, most of the results you see via the Insights tab will be about the overall user experience, often in relation to a site-wide goal. A page goal on the otherhand allows you to get more specific with a set goal for a single experience.
You start by creating a new on-site experience and once you are in the editor, you will see the option to Set up page goal on the right side, as marked in the image below.
After making the change(s) you wanted, you need to select an element on the page to track as the goal of your experience. For this, you can use the target button; click it once and after you are over the element you'd like to select, click again. That's it! Don't forget to save, create new version and publish, and start your experience.
To see the results of your experience(s) with a page goal, you can go to the Experiences tab of the Insights page and scroll down to Experiences with page goal. Here you will see the percentage of visitors in the control group as well as participants (visitors who have seen the experience) and the number of total conversions. You can also click Details to get more information.
Generally, testing your experiences is a good idea. However, running proper experiments requires a lot of traffic and the more experiences you test, the harder it gets to reach statistical significance. Also, the longer you run a test, the higher the risk of results polluted by external factors. Lastly, experiences influence each other, so with every additional experience it gets harder and harder to pinpoint what caused a dip or uplift in your goals.
Absolutely. You can run split tests, A/B/n tests, validate continuously with a control group, and make use of micro A/B tests too. The results are shown in your dashboard, in relation to the goals you've set.
The number of experiences (and thus tests) is unlimited, as well as the number of variations in an A/B/n test.
The control group helps you track what impact experiences have on your goals. Visitors assigned to the control group will see the original, unoptimized website. Navigate to the Insights tab to analyze how often they reach goals in comparison to participants (→see experiences).
Whether a visitor is a member of the control is decided upon their first visit and saved in a cookie. Changing the control group %, changes how new visitors are distributed between control and participants. However, returning ones are not reassigned. The numbers will realign over time.