A/B Test solutions for a locked source code environment - server side decides which user gets which variation

Question

I've been previously using tools such as Google Website Optimizer to run multi-variation or A/B tests. However right now I am looking for a solution that works for a larger site (400-500 000 unique visitor per month) with a very locked down source code environment. Basically:

1. The site is balanced over several servers
2. All code that is to be released on any of those servers must go via version control, unit testing and acceptans testing. All releases must be signed by develop, sys-admin and test executive.

This means that I am not allowed/it's hard to add "new code" (even if it's tested and verified) via Google Website Optimizer or any other of the GUI-paste-your-new-variation-here type of solutions.

We can however on server side decide which users gets which variation. Basically we can push the new version on X of the servers making 10-30% of the users view it for their entire session. The question is: Which tools do we use to measure "success" (i.e improved conversion rate). My idea so far has been:

1. Tag the new version in Google Analytics using a session variable (and then make reports based on segment) (similar to what is described on http://searchengineland.com/how-to-analyze-ab-tests-using-google-analytics-67404 )

2. Use Optimizely which has API support:

   window.optimizely = window.optimizely || [];
   window.optimizely.push(['bucketUser', EXPERIMENT_ID, VARIATION_ID])
   

What solutions have you tried for locked-down environments? Am I missing some obvious solution?

The site is in .NET/Episerver on IIS.

Regards, Niklas

Answer 1

We ended up going with Google Analytics and adding a session variable such as "abtest" with value "variation-4" and publishing it on certain nodes. It worked fairly well, with some limitations, namely that google analytics funnels doesn't have segment support.

Answer 2

You could use the AB-testing capability built into EPiServer CMO.

Answer 3

We did something similar and we have found Google Analytics documentation confusing. In the end the following code (made by server) got the work done for us:

<script>
  window.dataLayer = window.dataLayer || [];
  function gtag(){dataLayer.push(arguments);}
  gtag('js', new Date());
gtag('config', 'UA-xxxx-xxxx', {
  'custom_map': {'dimension1': 'abTestDesign'}
});

gtag('event', 'abTestDesign_dimension', {'abTestDesign': 0, 'non_interaction': true});
</script>

This code is generated by the server, where the last JS line is either that one of

gtag('event', 'abTestDesign_dimension', {'abTestDesign': 1, 'non_interaction': true});

Seems to be working fairly well on Numbeo.com