Most organizations will see at least a double-digit increase in their conversion rates using multivariate testing. So says a recent white paper by Ektron called "Fail Faster with Multivariate Testing." [pdf] Even a simplistic example of a company that realizes 100 conversions a day from a landing page bears this out. Say each conversion is worth $100. If this company runs a multivariate experiment and sees an increase of 10% in its conversion rate that will lead to a revenue increase of $1,000 per day, Ektron says.
In reality, though, getting to that point is not easy - there are a number of challenges and decisions a company has to make before it realizes a significant increase in conversion rates with multivariate testing.To succeed, consider the following:
1) Understand the business goals and problems you are trying to achieve, says ClickZ - and don't get focused on choosing the right software until that is settled. "To select the right one, you have to weigh your needs and your company's level of sophistication when it comes to running experiments against price, and keep in mind that the tool you start out with today may not be a good fit to solve all the challenges.
2) Choose the right software. This is not necessarily easy especially for small firms, which are faced with 20 or so packages on the market that could fit their needs. A free guide of these applications is available here.
3) Put a lot of thought into the test design. One decision - to name one example - that is not to be made lightly is whether to use full or fractional test designs, according to Billy Shih at Webtrends. Shih gives an excellent primer on the two approaches in his post - along with why he believes fractional is the best choice for multivariate testing of online campaigns. "Once you decide to test for all possible interactions, you are committing to a full-factorial test and incur the associated traffic requirements. I’d love to see a test design that is designed for full interactions and still makes sense," he says. "Not having the ability to reduce the number of interactions is a huge detriment rather than a benefit of solutions limited to full-factorial testing."
Other decisions that have to be carefully mapped, Shih says: Determining the number of factors and levels that can be tested based on estimated conversion traffic and deciding when to stop the test (when it has stabilized, not based on earlier estimations, Shih says.)
4) Never presuppose you know what your customers like, says Jim Keller, SVP of ShoeBuy.com. In other words, test everything. "What we have found is that when we guess what will appeal to customers we find out we were wrong through multivariate testing." Even small changes to a page such as a color or font size can make a difference in conversion rates, he tells MarketingVOX. ShoeBuy uses CoreMetrics, its own proprietary set of tools and Amadesa to support its testing program.