Traffic Reallocation

What is traffic reallocation and how does it differ from a simple traffic repartition reconfiguration?

In A/B testing, an important feature is the ability to reconfigure the traffic repartition (also called deviation) in a running test. This allows manual (or even automatic with algorithms such as Multi-Armed Bandits) control over the percentages of visitors seeing the different variations of the test, and for instance the winning variation can have its traffic percentage increased over time. Normally, a deviation reconfiguration is only applied to new visitors. Visitors previously bucketed into a test keep their first variation assignation throughout the test, because doing otherwise could result in the same visitor seeing several variations of a given experiment.

However, Kameleoon also supports traffic reallocation, which is the same as a traffic repartition reconfiguration, except it will apply to ALL visitors. In effect, this means any visitor that triggers the experiment will have a new variation assignment, regardless of what he may have seen previously. As you might have guessed, this can be a dangerous feature, but is also sometimes useful. This article explains how to perform traffic reallocation with Kameleoon and in which use cases you may want to use it.

How to perform traffic reallocation on the Kameleoon platform

The first step is to choose the new traffic repartition. This is exactly the same as a standard deviation reconfiguration. For more details about the use of the Kameleoon deviation configuration interface, please refer to this article in our user manual.

Usually the traffic deviation is locked, to prevent accidentally changing the deviation. After unlocking, it is possible to drag the sliders.

Once you have selected your desired traffic repartition, you then have to click the "Reallocation option" button, and you will be taken to the traffic reallocation dialog:

As you can see, you must select which variations the reallocation will be applied to. For every checked variation, previous visitors bucketed into this variation will be re-bucketed. Usually you will want to select all variations, but there can be exceptions.

For instance, let's say you have 3 variations: Original, A and B. You start the experiment with a repartition of 50% / 25% / 25%. Quickly you realize that variations A and B both greatly outperform the original, but it's not very clear which one between A and B is the best. You would then like to divert all the traffic from the Original variation into the two winning ones, even if this means performing reassociation for visitors previously bucketed into the original.

Reallocation allows you to do just that, without starting a new experiment and without disrupting the visitors that were assigned to A and B. Just select a 0% / 50% / 50% repartition, click on the Reallocation button, select only "Original", and validate.

Technical details and implementation of traffic reallocation

The bucketing of a visitor by Kameleoon (eg, which variation is selected and remembered for a given experiment) is persisted on the Kameleoon cross-domain local storage.

Current specifications are as follows:

Local Storage example:

kameleoonExperiment-72318: {"value":"210560/1542704630228/false","expirationDate":1545296630228}

Key used in local storage: kameleoonExperiment- + id of experiment
Value in local storage: {"value":"variationId/associationDate/internalBoolean","expirationDate":expirationDate}
Default expiration date: 30 days by default, but customizable

Note that in local storage, there is no "built-in" expiration date as when using cookies. This is why the value of the LS content is actually a JSON with two keys, "value" and "expirationDate". "value" contains the actual information to be persisted, while "expirationDate" corresponds to a (virtual) expiration date when this key would not been read at all. Our JavaScript engine automatically "emulates" the concept of expiration date with local storage keys.

The value in itself is made up of three notions, separated by slashes. The first number represents the actual id of the variation selected for this visitor (for this experiment). The second number represents the exact associationDate. The third value is a boolean that is used internally by our system, and which should be left alone.

When a user applies a reallocation to an experiment, the exact time of the reallocation is written to the variations that are affected. Note that we also have to write the time of reallocation to any part of the traffic excluded from the experiment.

Kameleoon.API.Experiments.getActive()[i].variations[j].reallocationTime: 1542979306144
Kameleoon.API.Experiments.getActive()[i].untrackedTrafficReallocationTime: 1542979306144

When a returning visitor is detected, the Kameleoon engine will check if the assignation time is older than the reallocation time. If this is the case, the user will receive a new variation assignment, according to the updated repartition of the experiment.

Implications of traffic reallocation vs. simple traffic allocation

Implications on the user experience

When doing a traffic reallocation, you will be showing some visitors a new variation, different from what they have seen previously. This can have serious implications on the user experience, but it depends mainly on the changes tested in the experiment. Imagine a test that would include different prices, offers or a different sales funnel. In a first visit, the user would have been made familiar with products and prices, and how to navigate to the checkout. In a second visit, the user would see a whole different configuration, a different checkout navigation, and could even understand that the prices have been changed. This could lead to a confused or even frustrated user.

If you're testing a whole redesign of a website, this can lead into a higher bounce rate, since the returning user is not recognizing the initial layout anymore and could think he is now on the wrong site.

These are extreme use cases though; most experiments do not consist of changes that would seriously impact the user experience. Also, remember that the exact same effects happen when you just stop a running test and implement the winning variation in your CMS (thus applying it to 100% of the traffic).

Implications on results analysis

Returning visitors usually have a much higher chance of converting (especially for businesses with long sales cycles, such as a website selling travel packages). For this reason, you have to be careful when performing a traffic deviation reconfiguration. Imagine an experiment with two variations, initially at 50% / 50% repartition. If you perform a deviation reconfiguration without reallocation after one month, switching for instance to 25% / 75%, the second variation will have more new visitors than the first one after the change. This means results will be biased in favor of the first variation, which will have more returning visitors in its overall tested population. If you wait long enough, the results will stabilize, but you must be extra careful in your analysis.

With a traffic reallocation, this problem does not occur since all traffic is rebucketed and the returning visitors proportion is unchanged by the new deviation (25% / 75%).

So reallocation usually has a positive impact on result analysis (but the bias induced by the user experience disruption is usually not worth it).

Use cases for reallocation: why would you make use of reallocation?

Traffic reallocation should be used with caution, since it can seriously affect the user experience. Nevertheless, there are use cases where it does make sense to use this feature.

Reallocation use case 1: elimination of a variation in an experiment containing more than two variations

One experiment with several variations reached a stable confidence level, and one of the variations clearly underperformed. You would like to get more insights from the remaining variations, and so the experiment should go on. In this case, it's useful to completely remove the low-performing variation: no visitors should see it anymore, even those who were previously assigned to it. With traffic reallocation, you can do that. By not selecting any other variation for reallocation, you will also ensure that visitors assigned to one of the performing variations are not rebucketed (which is what you want in this scenario).

Reallocation use case 2: switch to 100%

If you have found a winning variation, and want to shift 100% of the traffic to it, a reallocation makes sense in order to also switch the previously tested traffic that was assigned to other variations. If you don't do this, remember that a part of the traffic will still see the other variations.

Reallocation use case 3: very low proportion of returning visitors

Let's take a total website redesign, a standard A/B experiment with variation B being the redesign. There are many changes on variation B. It has thus been decided to send only 5% to the new variation at first to reduce risk. But if we have a low proportion of returning visitors, it can take a lot of time to later increase traffic to the new variation, once we feel ready for it. You could either setup some traffic as excluded from the experiment at first (keeping it available for later). Or, if you chose an initial repartition of 95% / 5%, and forgot to set aside excluded traffic, you could fix this by using a reallocation.

Traffic reallocation when using Kameleoon SDKs

When using our server-side SDKs (note: this does not include native mobile SDKs for Android or iOS), due to the implementation of variation assignment, any reconfiguration of the deviation will trigger a mandatory reallocation.