A/B testing is the most reliable way to improve your eCommerce conversion rate. Instead of guessing what works, you let your customers tell you through their behaviour. Yet many eCommerce businesses either do not test at all or run tests incorrectly, drawing false conclusions from insufficient data. This guide covers how to test properly at every experience level.
The Fundamentals
An A/B test splits your traffic between two (or more) variants of a page or element. Half your visitors see the original (control), half see the variation. After enough visitors have been through the test, you compare conversion rates to determine which variant performs better. The key word is "enough" — statistical significance matters.
Before running any test, formulate a hypothesis: "We believe that [change] will [effect] because [reason]." For example, "We believe that adding customer reviews below the product description will increase add-to-cart rate because social proof reduces purchase hesitation." Without a hypothesis, you are just randomly changing things and hoping for improvement.
What to Test First
Focus your testing efforts where they have the most impact — pages with high traffic and high intent:
- Product pages: Test product image layouts, description formats, CTA button text and colour, price display, and trust signal placement.
- Category pages: Test grid vs list view, number of products per page, filter placement, and sorting defaults.
- Cart page: Test showing vs hiding the cart summary, cross-sell placement, free shipping thresholds, and urgency messaging.
- Homepage: Test hero banner messaging, featured product selections, and value proposition placement.
Statistical Significance and Sample Size
The most common A/B testing mistake is ending tests too early. You need to achieve statistical significance — typically 95% confidence — before declaring a winner. This means the probability that the observed difference is due to chance is less than 5%. Use a sample size calculator before starting to determine how long your test needs to run based on your current traffic and conversion rate.
For a store converting at 2% that wants to detect a 10% relative improvement, you need approximately 30,000 visitors per variation. At 500 daily visitors, that is 120 days — two months per variation. If your traffic cannot support this, focus on testing higher-impact changes (larger expected effect sizes) or test more prominent elements where differences are more pronounced.
Tools for eCommerce A/B Testing
Google Optimize was discontinued, but several strong alternatives exist. VWO and Optimizely are enterprise-grade platforms with visual editors, advanced targeting, and server-side testing. Convert.com offers a privacy-focused option popular with European businesses. For Shopify stores, Neat A/B Testing provides a lightweight, Shopify-native solution.
Whichever tool you choose, ensure it supports proper revenue tracking — not just conversion rate. A variant might have a lower conversion rate but higher average order value, resulting in more revenue per visitor. Revenue per visitor is the metric that matters most for eCommerce tests.
Advanced Testing Strategies
Once you have mastered basic A/B tests, explore multivariate testing (testing multiple elements simultaneously), personalisation tests (showing different variants to different segments), and sequential testing (building on winning variants with further iterations). Server-side testing eliminates the visual flicker that client-side tools sometimes cause and allows testing of backend logic like pricing algorithms and recommendation engines.
Build a testing culture within your team. Maintain a test backlog prioritised by expected impact, document every test result (including failures — they are just as informative), and calculate the cumulative revenue impact of your testing programme. At Born Digital, we help eCommerce businesses establish systematic testing programmes that deliver compound conversion improvements over time.