Shopify A/B Testing: Native Rollouts vs Apps (2026)

Most Shopify stores ship changes on a hunch. A new homepage layout goes live because it looked better in the theme editor, not because it sold more. Shopify A/B testing fixes that by showing two versions to real shoppers and keeping whichever one performs reliably better. For years this meant installing a third-party app, since the platform had no built-in option and Google's free Optimize tool shut down in 2023. That changed with Shopify Rollouts, the native feature for scheduling and split-testing storefront changes. This guide covers what Rollouts can and can't test, when you still need an app, what to test first on a real store, and how to run a test that gives you a trustworthy answer instead of a false positive.

What Shopify A/B testing is, and what Rollouts changed

An A/B test splits your traffic between a control and a treatment, often 50/50, then measures which one converts better. The method is old. The native Shopify support is new.

Until recently, running an A/B test on Shopify meant a third-party app. The platform shipped no built-in testing, and the free option most people reached for, Google Optimize, was retired. Google confirmed Optimize and Optimize 360 were "no longer available as of September 30, 2023" (Google). That left merchants choosing between paid tools, custom setups, or guessing.

Shopify now has native A/B testing through a feature called Rollouts, found under Markets in your admin. A June 5, 2026 changelog update expanded it to schedule, publish, and A/B test new themes plus checkout and customer-account configurations (Shopify). It arrived as one of the Shopify Spring 2026 updates worth reviewing for conversion impact. You can run a real experiment between two versions of your store without writing code or paying for a separate tool. Going native also lets you avoid adding another storefront testing script to your store.

That sounds like it replaces the apps. It doesn't, and the difference is the whole point of this guide.

What Shopify Rollouts can and can't test

A rollout is "a scheduled set of changes to your online store's main theme, checkout and accounts pages" (Shopify Help Center). You can do three things with one: publish a change on a schedule, release it gradually to a percentage of visitors, or run it as an experiment against your live store to see which performs better.

What you can put through an experiment:

• A new theme, or an edited version of your current one
• A different checkout or customer-account configuration
• Localized content per market, like testing different CTA text by region
• How much eligible traffic enters the rollout, which Shopify calls the "launch reach"; the experiment then splits that traffic between the control and the treatment

You can run more than one experiment at once, and Shopify automatically gives each one a mutually exclusive slice of traffic, so a visitor only ever enters one rollout for a given resource.

The plan requirement is where most write-ups get it wrong. Plenty of agency posts say you need the Advanced plan. Shopify's own documentation is more generous.

Key takeaway: Scheduling and gradual rollouts are available on the Basic plan or higher, but running an A/B experiment needs the Grow plan or higher, per Shopify's documentation. Guides that say "Advanced plan" are out of date.

Here is the part most write-ups get wrong. A Rollout treatment can be a single change you make in the theme editor, not only a whole redesign. You compare a control against a treatment at the theme or checkout-configuration level, and that treatment can be one edited section, different CTA text, or a full theme replacement. Shopify's own examples include testing different CTA text per market and a new upsell block in checkout.

What Rollouts genuinely can't do:

• Price, shipping, or offer tests, since a price is not a theme or checkout configuration
• Liquid template edits inside an "edit main theme" rollout, since Shopify blocks Liquid changes on that path; to test code-level changes, build a separate theme and run it as the replacement theme. Rollouts also don't run on vintage themes (Shopify Help Center)
• Behavioral audience targeting, like showing a variant only to returning visitors; targeting is built around Shopify Markets, not visitor segments
• Custom goals or deeper statistics, since the experiment metrics are fixed

None of that is a knock on Rollouts. It runs theme and checkout experiments well, with no separate testing-app fee on eligible plans. It just leaves a gap the app ecosystem still fills.

Native Rollouts vs a third-party A/B testing app

Short version: use Rollouts for theme and checkout tests, and reach for an app the moment you need to test a price, a specific audience, or a custom goal.

The apps each lean a different way. Treat their capability claims as what each vendor says it does:

• Intelligems is built around price, shipping, and offer testing, the "find the price that maximizes profit per visitor" angle (Intelligems). This is the clearest thing Rollouts can't do.
• Shoplift focuses on no-code testing of themes, templates, and individual sections, down to a single block (Shoplift).
• Visually.io adds segmentation and personalization across the funnel, from the homepage to post-purchase (Visually.io).
• Shogun previously included A/B testing inside its page builder, but since November 12, 2025 that integrated feature is limited to existing eligible Shopify users; new users are directed to Shogun's standalone A/B Testing app (Shogun).
• Convert leans on deeper statistics and specific audience conditions for stores that have outgrown basic reporting (Convert).

For a store on the Grow plan that mostly wants to compare layouts and checkout setups, Rollouts is enough on its own. An app earns its monthly fee once you need capabilities outside Shopify's supported resources, such as price and offer testing, behavioral targeting, custom goals, or deeper reporting.

You can also run Rollouts and an app side by side. A common setup keeps native experiments for theme and checkout changes and adds an app like Intelligems only for the price tests Rollouts can't touch. One caution: Shopify keeps its own rollouts mutually exclusive, but that protection does not extend to a third-party app, so avoid running an app test and a Rollout on the same page or funnel stage at once, or the two treatments can collide and muddy both results.

What to A/B test first on your Shopify store

Test where the traffic and the money are, not where the change is easiest. A homepage hero is highly visible to the people who land on the homepage. A footer link gets far less attention.

A workable way to rank ideas is the PIE framework, scored on Potential, Importance, and Ease, from WiderFunnel founder Chris Goward (Growth Method). High-traffic, high-impact, low-effort ideas go first.

There is no single right first test. Weigh ideas by impact, traffic, evidence, and effort. A rough starting order for many stores:

1. Checkout and cart, because that is where the money leaks fastest. Baymard puts the average cart abandonment rate at around 70% and estimates the average large ecommerce site could increase conversion by around 35% through better checkout design (Baymard). On Shopify, cart and storefront changes made in the theme editor run as theme experiments, while checkout changes are tested by comparing checkout configurations; pricing, behavioral targeting, or custom metrics may still need an app. Whether a cart drawer or a full cart page converts better is a classic test to start with. Our guide to customizing Shopify checkout covers what is editable on each plan.
2. Product pages, where the buying decision happens. Images, the title, review placement, and especially add-to-cart button copy and placement are the usual suspects.
3. The homepage hero and the promise above the fold, since it sets the first impression for the most traffic.
4. Navigation, search, and collection layouts, which matter more once the pages above are solid.

That order is a starting point, not a rule. Checkout carries the highest value per visitor but also less traffic than the homepage or product pages, so a smaller store often reaches a reliable sample faster by testing those first.

Picture a concrete case. An audit flags that your product page buries the add-to-cart button below a long description. The hypothesis writes itself: move the button above the fold. If the change lives in the theme editor, a Rollouts experiment can test it directly; an app helps when you need unsupported targeting, custom metrics, or a dedicated visual workflow. Either way the test tells you whether the change earns more orders or just feels tidier.

The audit angle ties this together: a store review flags a weak spot, and an A/B test confirms whether your fix actually moves the number. For a sense of where typical stores fall short first, our Shopify CRO benchmarks from 190 audited stores show the patterns that repeat, and the conversion rate benchmarks by industry give you a baseline to beat.

How to run a test that gives you a real answer

A test only helps if you can trust the result. Three rules keep you honest.

First, pick your sample size before you start, and don't stop early. Evan Miller's "How Not To Run An A/B Test" shows why this is the rule that matters most. If you watch an experiment and stop the moment it looks significant, your real error rate balloons. Checking ten times turns what looks like 1% significance into about 5%, and a run-until-it-wins approach can reach "26.1%, more than five times what you probably thought" (Evan Miller). Decide the number of visitors up front and wait for it.

Second, when your testing platform reports statistical significance, many teams hold results to a 95% confidence level (CXL). That is not a promise that a winner has only a 5% chance of being wrong; it means the test uses a method built to keep false positives rare. Shopify documents fixed outcome metrics for Rollouts rather than a built-in significance threshold, so work out the sample size you need before launch and run a separate statistical check when you want one.

Third, run for full business cycles. A weekend shopper behaves differently from a Tuesday one, so two full weeks is a sensible floor, and lower-traffic stores often need four or more. VWO frames test length as a function of sample size rather than calendar days (VWO).

Reading the results comes once the test is running. Open the rollout under Markets, then Rollouts, and Shopify shows the experiment's analytics: for a theme test, conversion rate, bounce rate, reached-checkout rate, and add-to-cart rate, while checkout tests add the checkout conversion rate (Shopify). Two caveats matter. Rollouts reports these numbers but doesn't crown a winner or show statistical significance, so weigh them against the sample size and threshold you set up front. And the analytics only collect when you run the change as an experiment, not when you publish it to everyone at once.

Key takeaway: A/B testing only works once you have the traffic to fill a sample. Below that, testing low-traffic pages produces noise, not answers, so improve them with proven patterns and save testing for your busiest pages.

As a rough gauge, a store converting at around 2% needs far more visitors per variant than one converting at 4% to detect the same relative lift, which is why low-traffic pages rarely reach a verdict.

You can sanity-check whether you have the numbers with an A/B test calculator before you commit a single visitor to an experiment.

FAQ

Does Shopify have native A/B testing?

Yes. Shopify added native A/B testing through Rollouts, found under Markets in the admin. It can experiment between two themes or two checkout and customer-account configurations and split traffic between them (Shopify). Before this, A/B testing on Shopify needed a third-party app.

Is Shopify A/B testing free, and which plan do I need?

Rollouts is included with Shopify, but the experiment feature has a plan floor. Shopify's documentation states experiments are available on the Grow plan or higher, while scheduling and gradual rollouts work on lower plans. There is no separate per-test fee.

What can Shopify Rollouts test, and what can't it?

A Rollout can test a single theme-editor change, like different CTA text or one section, or a full theme, plus checkout and customer-account configurations and localized content per market. It cannot edit Liquid templates inside an edited theme, though code-level changes can run as a separate replacement theme, and it does not support vintage themes. For product prices, shipping rates, behavioral audience segments, custom goals, or deeper reporting, you need a dedicated app.

Shopify Rollouts or a third-party app, which do I need?

Use Rollouts when you want to test theme-editor changes or checkout setups without a separate testing-app fee. Choose an app like Intelligems or Convert when you need to test pricing, behavioral audience segments, or custom goals, which Rollouts does not cover.

Can you A/B test pricing on Shopify?

Not with native Rollouts, since it tests theme and checkout configurations rather than prices. Price testing on Shopify is handled by apps built for it, such as Intelligems, which tests prices, shipping rates, and offers (Intelligems).