Multivariate testing CMS driven websites

Website multivariate testing is increasingly common as companies look to optimise website performance. However, multivariate testing for websites with Content Management Systems (CMS) is far from straight-forward.

What is multivariate testing?

For those who are not familiar with the concept of multivariate testing, the basic principle is to have two (or more) separate variants of the same page on a website and track click-through/conversion on each variant of the page separately. This allows a website manager to evaluate the effectiveness of a particular change to a page vs. the previous version of the same page. For example; it will provide definitive answers to the questions “has the change improved conversion rates” and “has the change improved click-through ratio”.

This effectively allows some form of “trial and error” optimisation whereby changes can be piloted and then either made permanent or revoked depending on their impact. Google Analytics provides great facilities for split testing pages linked to pay-per-click (PPC) marketing campaigns.

What are the challenges when multivariate testing in a CMS?

Most website managers will now be considered at somewhat of a handicap if they are unable to take advantage of multivariate (or split) testing tweaks to pages. The principles behind a CMS mean that the structure of a page is normally defined in a template and the content (editable by CMS users) is rendered within this set structure.

So is it possible to retain a relatively transparent experience for CMS content authors whilst facilitating split testing? I think the answer to this question depends on the flexibility of the CMS you are using.

How can you overcome these challenges?

In my case I was looking at the Umbraco open source ASP.NET content-management system and I was able to provide a solution, but only by adding some custom code and replacing the standard Umbraco default page. From version 4, Umbraco has used ASP.NET masterpages as the underlying technology behind it’s page templates. Each page entity must be tied to a single template and therefore a specific masterpage file.

For example; in the custom version of the Umbraco default page that I created the OnPreInit event of the ASP.NET page lifecycle was handled; and within the handler the MasterPageFile property of the page was set dependant on a querystring argument. Therefore regardless of the template that was assigned to the page within the CMS I was able to apply whatever masterpage file I wished. After overcoming this hurdle it was relatively easy to create a macro within the CMS to allow users to create separate content for the A, B or C variants of each page as they desired.

Thomas Coles co-founded MSM in 1998 and is the largest shareholder with 44.7%. His key achievements so far include growth from 2 to 40 FTE; high levels of customer satisfaction and retention, as well as surviving the sector downturn from 2001-2003 and growing the business in the 2008-2009 recession. Thomas’ business acumen was apparent from a young age. As a child (aged 8) he was already budgeting his pocket money on a spreadsheet. His passion for technology was also evident, as, aged 10 he was writing programmes for his Amstrad. Thomas started the MSM business soon after graduating with his father, who remains a non-executive director today. A strong believer in applying common sense to any situation, Thomas says his objective is to continue to be criticised for being too honest. Away from the office Thomas enjoys family life with his wife and three children and likes to take part in half marathons, going to the gym and watching Formula 1 motor racing. Thomas is also a trustee of a local charity.