Why 9 Out of 10 A/B Tests Fail (and How Shopify Brands Pick the Ones That Win)

Steve Hutt: Hey there. Welcome back to eCommerce Fastlane. I’m your host, Steve Hutt. Today’s conversation is all about testing before you commit. I’d argue that a lot of brands aren’t consistently testing. There’s that saying: if you’re not testing, you’re guessing. Sometimes people go back and forth over a price increase, a free shipping threshold, or a new landing page. In my experience, the final call usually comes down to a gut feeling, personal taste, or what you feel as the founder or marketer. I’d argue that’s the incorrect way to think about it. Having taste and ideas is great, but looking at concrete numbers is incredibly important. That’s why I have my guest on today. It’s Jeffrey Lin, founder and CEO of AB Convert, at abconvert.io. One unique thing about their company is that they have the Built for Shopify designation. There are quite a few hoops to go through from a privacy and data-handling perspective to earn that. They’re an A/B testing app, and I believe a couple of thousand stores now use their solution for price testing, shipping offers, and pages, all the things I mentioned. It’s pretty cool. Jeffrey, I looked at your LinkedIn profile before recording today, and you started from your parents’ house. We’re talking bootstrapping, no funding at that point, and no real ecommerce experience. You have that experience now because of this business, but you bootstrapped everything from the beginning. I’m always excited about people who dig deep and passionately want to help others. I’m hoping we can learn what’s winning today in testing and experimentation, and come away with some profitable ideas for improving a site. Jeffrey, that was a long intro, but welcome to the podcast.

Jeffrey Lin: Hello, everyone. I’m excited to share, and hello, Steve. I’m grateful to be on the podcast.

Steve Hutt: Absolutely. Take me back. I know a little about your backstory, but you had a conversation with a classmate that led you to build your first Shopify app. What problem couldn’t he solve that you decided to solve?

Jeffrey Lin: It was quite interesting. About four years ago, my classmate and I were two young guys, around 22 or 23. He was already a successful dropshipper, and I watched him grow his store from the dorm. I learned from him, too. I launched my own dropshipping store, watched the Ecom King on YouTube, and tried to follow the course and launch a site. One thing we didn’t have a clear process for was pricing products. My friend uploaded a few hundred to a thousand SKUs per day, and he always decided the price by applying a formula based on three times the cost of goods sold. Sometimes he couldn’t tell whether the problem was the ads or the price point. It was all art and not scientific at all. He told me he wished he had a way to figure that out. At the time, he had a few brands that relied on influencer marketing rather than Meta ads, so his margins varied by influencer. He wanted healthier profit margins to sustain future growth. I was chatting with him and said, “Tell me more,” because I didn’t understand how ecommerce worked. He gave me some suggestions and told me to look through the Shopify documentation. I was still searching for a startup idea, so I thought it might be worth looking at the docs and thinking about what could solve this pain point: having a better way to launch new products and get campaigns going. That was the start. It was very early. I had no ecommerce experience, I wasn’t in the industry, and we were in Taiwan. Neither of us had worked before.

Steve Hutt: It’s so interesting. I always love hearing these stories. If you’re entrepreneurial, you want to be in the startup world, but you don’t necessarily know what problem you want to solve. Being open-minded and listening to what others are talking about can reveal an opportunity. Even though you had no ecommerce experience, your friend clearly did, being in the dropshipping world and learning from YouTube. Suddenly, there’s an opportunity, and that’s why you’re here today. I want to talk about some of the tests you’ve run. I think you talk about this publicly, but what share of these tests actually beat the original? I always wonder about that. Someone might say, “I’m testing, but I believe where I am today is probably the best spot from a profitability or lifetime value perspective.” Your data shows something interesting. Of all the tests you run, how many actually beat the original?

Jeffrey Lin: It’s an interesting question because there are different ways to look at the probability of beating the original. Preparation makes a difference, and the test type makes a difference. From a high-level perspective, we first exclude tests that stop early. A lot of people launch a test, find an issue, and stop it. Removing those outliers, about 10% of tests beat the original, but it varies by test type. That’s related to what I’m always saying: most of the success lies in how you research and prepare your test. We have price tests, shipping tests, landing page tests, content tests, and checkout tests. They all have different probabilities of winning. In general, I would say 5% to 10%, but mostly around 10%. That means you’ll have nine failed tests out of every 10 you run. That’s for brands testing consistently. For people who aren’t testing consistently, the results are much more random because, in the first couple of tests, people don’t necessarily know what they’re doing. Markets also differ. Sometimes the probability is lower, and sometimes market conditions are better. In short, it’s roughly 10% on average.

Steve Hutt: That’s amazing. There are a couple of things I want to talk about because there are lots of different tests people can do. I want to make sure we’re educating even those who are at an early stage. Pricing is an interesting one, and we’ll get into that in a minute. Shipping is particularly interesting because a lot of buyers understand that if they spend a little more, they’ll hit the free shipping threshold. That’s part of the contribution margin you’re making. There’s a shipping cost based on your product, but as you increase the basket size, there’s enough net profit in that bundle or total basket to offer free shipping. That motivates people. Instead of going into a store, they can say, “I can buy two and get free shipping.” Talk about that, because I think a lot of people look only at their competitors instead of looking at their own data to justify where their free shipping threshold should be.

Jeffrey Lin: Free shipping thresholds are an area where we’ve seen a lot of success, and shipping-related tests generally have a higher success rate. It’s an interesting test type to consider. I sometimes compare testing to flipping a coin, but this is more of an asymmetric bet. If you increase the threshold, customers have two options. If they don’t reach it, they pay for shipping. If they do reach it, they’ve spent more on the basket to get free shipping. Either way, there can be a net positive. The question is whether people drop off and don’t check out, which is what we need to watch. When executing a free shipping threshold test, the messaging is important. Most people update the progress bar and the banner on the website, but some don’t even do that. How you communicate it affects the outcome and the conversion uplift at checkout. If you already have a threshold set up, that’s the easiest starting point because you have existing order data to analyze. Look at the order distribution and what a typical basket looks like. Is it a main product plus an accessory, or is it just one product? You can export your orders and put them into Claude to explore what threshold makes more sense. You want to distinguish between customers willing to spend more and customers who only want one product and won’t meet the threshold anyway. You want to draw a clear line between those two groups. For example, if your threshold is $75 and your product is $60, but you don’t have a lower-priced product that lets someone reach the threshold easily, then a $75 threshold and a $100 threshold might not make a difference. Either way, the customer needs two items in the basket. Dig into your order data, average order value, and individual product prices. You can export that data from Shopify or connect Shopify through MCP and ask Claude to analyze where you could optimize the threshold. That’s how we suggest users approach it now. Previously, we had more of a fixed rule, but now there is more data to look at and connect. Let Claude help with the analysis. The idea is to make this a test with a strong chance of winning because you’ve done the analysis correctly.

Steve Hutt: There’s a store where I buy bar soap called Duke Cannon. It’s a men’s soap company, and they sell these big bars of soap. Their product pricing is around $7 to $9 each, and they have a free shipping threshold. I’m in Canada, and right now, shipping to me is $19. That’s about three bars of soap just for the shipping cost. Suddenly, the value isn’t there. It motivates me to add eight or nine bars to the cart to get that $19 shipping charge waived. I know it works because I’m a consumer who does that. Do I really want nine bars of soap? Probably not. But I also don’t want to pay $19 for shipping because I feel like I’m wasting two or three bars of soap if I buy less and pay the shipping charge. I’d rather pay a little more and get free shipping. You do that math in your head as a buyer, and I’m a case in point.

Jeffrey Lin: Exactly. That’s why I say there are differences between test types, but there are also seasonal differences. In our first two years, free shipping threshold tests and shipping tests weren’t that popular. Since inflation, people have become more conscious that products and shipping can be expensive. They’re more attracted to free shipping thresholds and free offers because the market has changed. How you operated last year doesn’t guarantee the same performance if you run the same test with the same setup this year.

Steve Hutt: Another thing you have in AB Convert is theme and page testing. It’s interesting because a lot of Shopify marketers run paid ads through Google or paid social, and they often find that conversion rates are better when they send people to a custom landing page, particularly for their top 10, 15, or 20 SKUs. I believe it’s important to tell a better brand story on a long-form product page rather than just a standard PDP, or product detail page. I think that’s been demonstrated. If you test an ad going to a PDP against an ad going to a landing page, I think the landing page usually wins in most cases. In your case, you can run theme, page, or landing page tests based on where the ad is coming from. I’m not sure how you distribute the traffic, but you can pick a winner after a certain period. There are different schools of thought about whether to create a landing page. With AI tools and AI page-design tools, you can create two amazing options, but the tool can’t tell you which one will win. It’s great for the design part, but then you pass it over to AB Convert to implement the test and help identify a winner.

Jeffrey Lin: With theme tests, page tests, and landing page tests, the intent is the same: how can I get my top-performing page to a 3%, 4%, 5%, or even 7% conversion rate? Whether you’re testing blocks within the same page template or comparing different formats, such as a long-form product page, a listicle, or a standard PDP, you’re trying to optimize landing page conversion. When it comes to execution, I always say you need to know where the problem is. Focus on your top-performing landing pages and understand the conversion rate per page, not just for the store overall. There are several ways to do that. We can help analyze it, or you can connect your Shopify store to Claude and use analytics skills to get the conversion rate for each page. This is important. Let’s say you have five landing pages and they’re all converting at 2%. That may tell you that applying the same format isn’t going to produce a major improvement. If you want to unlock 2.5% or 3%, you may need to change the format rather than add a small review block or make a minor tweak. It’s not necessarily a wrong setup, but you could try a different setup. In that situation, consider a URL redirect test. It’s very convenient now. I’ve seen users use page builders, Replo, or Claude to create pages. You can vibe-code a page and create a redirect test, and the whole setup can take about 15 minutes. It’s very efficient. That’s useful when you can’t identify a small improvement that would move the needle. You may need a bolder attempt. If all your winning ads have always landed on PDPs, try a listicle or long-form landing page that’s completely different. People often think about landing page optimization only through the lens of design and offers, but it also comes down to how you set up your ads. Connect those pieces and take a holistic view. Make sure the messaging, offer, and criteria in the ad match the landing page. Get the angle right, then test different formats. If you do see variation in conversion rates between pages, dig into the design. Watch heatmaps, look at clicks around the buy box, and identify clusters of activity. If a secondary image is getting a lot of clicks, perhaps enlarge it or use a two-by-two image layout. You might move the image slider higher or bring a section with a lot of engagement closer to the top. There are many strategies, but the most important thing is to find the big gap. People are busy, so focus on an opportunity that deserves your attention. You want to hit a home run, not just get to first base.

Steve Hutt: That’s amazing. I know Shopify has what I believe is a native A/B testing solution called Rollouts. I think it handles themes, maybe some checkout configurations, and possibly discounts. For a merchant listening who wonders whether they need another app or whether Shopify’s native option is enough, when does AB Convert become a better fit?

Jeffrey Lin: I’m familiar with what Shopify Rollouts is doing, and I think it’s a great tool. When I started four years ago, there wasn’t anything like that. In our first two years, we were filling the infrastructure gap so people could run basic tests. Shopify Rollouts now has theme tests, checkout tests, discount tests using discount functions, and other configuration tests. Those are generally variable changes. The name “Rollouts” reflects the scenario it supports: someone wants to change something. For example, when you update a theme, you can create a rollout. It supports a continuous loop of updates. Your developers make changes, and you want to know how to release them safely. That’s the value proposition. For us, it’s more about someone saying, “I want to improve my business and scale. I could use Rollouts for the release, but I want to build an operating system and testing fundamentals, collect data, and better understand my audience segments and their performance.” They want a testing roadmap, advanced dimensions and attributes from different channels, more specific significance analysis, and follow-up tests. We’re focused on building that testing roadmap and the internal operating system for conversion optimization. That’s why we have so many connected pieces, including MCP, a visual editor, and different tools serving the same purpose. For content alone, we have theme tests, a visual editor, template tests, and URL redirects. We also support a feature where you can comment in the visual editor, send it to AI, and come back with a design. That’s the differentiation. I believe Shopify will do a great job and support more configurations, which we would call test types. We’re building an end-to-end solution from ideation through testing, follow-up, and rollout, along with ongoing personalization.

Steve Hutt: That’s amazing. I looked at it in my test store a couple of days ago. I think it’s under Markets in the admin, alongside Catalogs and Rollouts. It’s worth having a look. I think you can also use Sidekick, Shopify’s AI agent, to help. You bring up a good point about an MCP connection because a lot of mid-market or more advanced brands are making Claude their source of truth and connecting multiple MCPs. It’s interesting from a design perspective. You can use design capabilities within Claude to help build landing pages, then push those out to AB Convert or other tools. That gives it context around the business and a shared data pool: what are we pushing out, and what are we learning? That information comes back into Claude, builds its memory, and helps you decide what to test next. You’re the execution layer, while Claude can serve as a contextual brain in the middle, not just for A/B testing but for different parts of the business. It’s similar to what Sidekick is trying to do with Shopify, but you can take it further when you run your own ChatGPT or Claude setup, right?

Jeffrey Lin: Yes. We always try to think from the brand’s perspective, not just about our own benefit. Internally, we debate how to make the brand’s experience better as Shopify improves Rollouts. What’s the difference between our offering and using Shopify’s native solutions? I think Sidekick is great. It supports different intents and actions, and for brands that aren’t deeply focused on AI, Shopify can probably make Sidekick work for most of their needs. We’re looking for the gaps it doesn’t cover. When you’re building an app-embedded or software-embedded solution, there’s a context gap, as you mentioned. If you’re using Shopify admin, you want all the context from Shopify. That’s possible, but running a brand isn’t only about Shopify. You have Triple Whale, Facebook ads, and many other pieces of data to sync. At this point, brands often build their own agent stack and data sources. They might put information into a folder and give Claude, Codex, or another agent access to it. We serve as a connectivity layer for experimentation. Over time, we want to become more of an experimentation harness. We don’t really care who implements the experience. We want people to run tests correctly, build good skills and operating guidelines, launch safely, review and interpret results, and then launch another test. Another example brands might recognize is Higgsfield. It offers image generation with many underlying models, but what Higgsfield provides is a strong editing experience. That’s the direction we’re pursuing. We want Shopify Rollouts to be one of the execution layers. Small businesses can use Sidekick, while power users can use different tools. Maybe in the future, we can integrate with our competitors, too, so people can choose. It’s a world where everyone can win.

Steve Hutt: That’s amazing. It’s funny you mention Higgsfield because I use it myself. Claude is my source of truth. I’m on the media side, so I use it for content production, gap analysis, and understanding what else I need to write about. At the end of the workflow and research phase, when I’m ready to publish, I push information to Higgsfield with visual brand guidelines and have it create the featured image for the post. That comes back into Claude, and then it publishes to WordPress. There’s a loop, with checks and balances along the way to make sure everything links correctly. It’s pretty cool. I’ll include Higgsfield because it’s a neat tool for anyone listening who’s thinking about content and how to scale it. I’ll probably link to a post about what I do on the media company side, which I think applies to brands that want to build content and community around their business. Another thing I want to discuss is BFCM, because it’s on a lot of people’s minds. We’re not in a code freeze yet, so there’s still an opportunity to do things. Most people have probably thought about what they’re promoting. They’re locking in pricing and getting organized. Their products have arrived from China or wherever they’re manufactured, so they have what they have. You have a BFCM CRO playbook on your website. I haven’t read it yet. I downloaded it this morning and will go through it after our recording. From your perspective, is now still a good time to test a few things leading into the busy period?

Jeffrey Lin: Yes. To give you some context, the playbook was created by five partners, including us. It’s a summary of experimentation insights from different types of partners, and it covers principles for setting up your BFCM schedule. It was originally designed for eight weeks. Those weeks are divided between big-swing tests, smaller hypothesis-validation tests, and more exploratory tests. There’s a structure to the schedule. Now that we’re about four weeks into that and have four to five weeks left before BFCM, we can trim those elements into a more focused schedule. You can take an emergency path where you only have one or two choices to run. So yes, you can still test before the code freeze, but you need to be strategic about prioritization and the outcome you want. Imagine you only have time for one test. You can’t chase all the fancy tips from the internet. You might have a million ideas, but you have to pick one that addresses your biggest uncertainty. If you’re still struggling with the angle of your ads, run a URL redirect test. You might structure one campaign with a couple of tests over one or two weeks to close that gap. If you don’t know what offer to use or what discount percentage to run this year, test 20% versus 25% with the merchandising. If you don’t know your free shipping threshold or how to structure your shipping offer, test that. Work backward from the most uncertain part of your BFCM plan and run that exact test. Don’t be greedy. Pick one test and save the others for next season or the coming months. Once you’ve gained that insight, be happy with it and get back to preparing for BFCM.

Steve Hutt: It’s interesting to think about the traffic that’s happening now. Amazon had its Prime days, and I think it decided to go a little early this year. Some share of wallet has already been spent there. Amazon did billions, and people only have so much money to spend over the holidays. But there’s still a massive opportunity. People don’t buy only during Black Friday and Cyber Monday. Yes, a lot of money will be spent then, but people are buying now because the selection is good. Later, it might not be as good. Some brands may be wondering whether they should offer bigger discounts or test other offers. Bundling comes to mind. That’s separate from the free shipping threshold conversation, but are there ways to test offers other than taking money off a product? Or should they test smaller discounts? You sometimes wonder what taking 40% off means for a brand. Some brands don’t take 40% off. Some don’t discount at all. Brands like Lululemon, Vuori, or Alo Yoga may offer something like a free tote bag during the holidays rather than a discount. What’s your perspective on that? Why might brands not need to discount as much as they think? And how does your tool help them understand how much discount translates into revenue and profit?

Jeffrey Lin: I see each test as a lever or a tool. There’s no single fixed type of test you should run. If the question is, “How do I know whether I’m offering too much discount?” you first need to consider your operational constraints. A brand like Alo Yoga might not be able to run a price test, so it may only be able to change the discount or the format of the discount. Discounts take many forms: buy one, get one; a fixed discount; a progressive discount shown through a progress bar; and other options. Look at your constraints because discounts, shipping, and pricing all come back to profit margin. Understand those constraints before deciding on a strategy. That’s why I don’t simply give people a test to copy. I explain the logic behind why we’d choose it. Let’s say a brand is struggling with discounts and is worried about uncertainty, including whether demand around Prime Day is smaller or whether results are repeatable. To understand whether something is repeatable and measure the true outcome, start by challenging where your current strategy came from. Brands often repeat the same promotional strategy. Go back to the data. If you’re offering 40% off and want to know whether you can do better, look at gaps in the order distribution. For example, if you see a gap between $75 and $125 in average order value, that range may be an opportunity to use a discount to encourage a higher basket. You might test a 10% or 20% incentive. Execution matters, too. Consider a progress bar or a pop-up discount. If there are constraints on changing the headline number, you could add a free gift with a pop-up signup or use a progress bar. For BFCM, though, you want to simulate the actual day, which is why I encourage people to leave enough time to prepare their testing schedule. Setting lead collection aside and focusing on BFCM itself, you could use a progress bar, an add-on discount, a checkout product offer, or an exit-intent discount. Look at the drop-off points and add a recovery offer rather than making the discount the main driver. For the main offer, people are aware of discounts and compare-at prices. You might test the compare-at price presentation. It changes the psychological framing, even if you’re selling at the same final price. For example, if you’re selling a $100 product with a $200 compare-at price, you might test a $300 compare-at price without changing the selling price. That’s another strategy. To work through all of this, first look at your constraints, then your data, and then the time available to execute. At this point, the clock is ticking. You may have one choice, so you need to be clear about why you’re choosing it.

Steve Hutt: It’s interesting because you often hear these BFCM postmortems where someone says, “Our revenue was up 50% over last year.” But then you look at the contribution margin and the actual profit from all of that. A lot of brands talk about same-store sales being up without talking about what the P&L says. That’s what’s interesting about how AB Convert can work. You can think about all these variables. It’s one thing to increase conversion rate and turn more visitors into buyers, but at what cost? If you test how much discount to offer, you may find that a larger discount raises conversion but drives contribution margin way down. You might prefer fewer customers with more profit rather than more customers with less profit, especially if net profit is significantly lower. It’s a fine line that you need to figure out.

Jeffrey Lin: Totally. That’s why we focus on helping people understand the data and making that data available in Claude. We work on analytics, have an analytics MCP, and support many dimensions. We also work on integrations with tools such as Clarity, heatmaps, Google Analytics, Shopify’s own analytics, subscription tools, Recharge, Skio, Klaviyo, segmentation, and customer surveys. People can bring those integrations and our data into Claude, analyze them, and add their constraints. Brands know their constraints best. They can provide those constraints with a specific prompt, such as, “Help me design the right test. I only have one shot before BFCM.” Inventory is another factor to consider. In the past, this kind of analysis was a headache. It wasn’t something one person could do quickly. It took a team, a lot of assets, and data queries. Now you can connect the information. Most importantly, once you reach a conclusion, you have an actionable item: a well-designed test with the specifications you need. You can run that through Claude as well.

Steve Hutt: That’s amazing. I went through your agent-related information, and you have price, shipping, and offer tests; URL redirect tests; template, theme, color, and checkout tests; audience targeting; and personalization tests. We didn’t dig too deeply into audience targeting, including internationalization and UTMs, but that’s interesting. It’s also nice that you work well with other partners. When I was at Shopify, we’d recommend a solution, or I’d bring a podcast guest’s solution to my merchants and say, “I interviewed this person this week. Here’s what they offer, and I think it could be valuable for you to try.” The first question would be, “Does it work with my existing tech stack?” They don’t want more complexity. You’ve been intentional about that. With your own MCP, someone can keep Claude as their source of truth while the execution happens through AB Convert, which is great.

Jeffrey Lin: Yes, totally.

Steve Hutt: That’s amazing. How do people get the BFCM guide? We’re going to wrap up, but it’s more than 90 pages, so it’s a significant piece of content to offer for free, with contributions from four other partners. I skimmed it while we were chatting, and there’s a lot in there. What do you think the next steps should be for people listening today?

Jeffrey Lin: I’m not the kind of person who just wants people to install the app and become customers right away. I want to give value to listeners and brands. We worked on this playbook for a long time, so download it and take a look. We’ve included CRO principles, interviewed the partners, shared their quotes, and analyzed their past tests. Different partners have different specialties. Some specialize in landing pages, some in ecommerce management, and some are full-service Shopify partners. It’s a good resource. Secondly, if you haven’t connected Shopify to Claude or Codex, go into the settings, find the relevant plugin, connect your store, and do a one-time audit. As you approach BFCM, you first want to make sure you’re not making a mistake. Then you can think about how to win. People should rely more on tools and data. Forget about A/B testing for a moment. I’m not trying to sell anything. When you dig into your own data, you find problems, and those problems lead you to solutions. Whether you should add subscriptions, a bundle, or a review app, the information is in the data. If you’re still curious about what we’d recommend, we offer a strategy call. We’ll talk about your business and data and advise you on the next step. You don’t have to use us. If you like us, you can, but if you already have a plan and your own ideas, find the tools that suit you.

Steve Hutt: That’s amazing. Looking around your website, I also see that you have a Claude skill people can download to help understand and visualize their site. If you’re a Claude user, go to abconvert.io. That’s a good starting point. I saw the BFCM guide pop up in the lower-left corner, and it’s also under Resources, so you can download it there. I think this will be an impactful episode because the timing is still right. As you said, we’re not in a code freeze, and there’s an opportunity. You can improve without driving incremental traffic to your website. There are ways to improve conversion rate or profitability through testing. There are lots of things you could test, but pick one or two and run those early tests. Traffic may double or triple during BFCM and into the rest of Q4, so testing now gives you something to work with as you head into that period. Thanks, Jeffrey, for recording today. Our time zones are quite different. I’m in Vancouver, and it’s morning here, while it’s nighttime for you. I appreciate you coming online to chat with me, and thanks again for helping Shopify brands.

Jeffrey Lin: Thank you, Steve, and thanks to your audience. Thanks for having me. I’m always happy to share.

Steve Hutt: Have yourself a great evening. Thanks so much.

Jeffrey Lin: Thanks. Bye-bye.