Before you pay anyone to optimize for AI, find out whether your own site is turning the AI crawlers away. For two weeks, ours refused more than eight in ten of PerplexityBot’s requests.
Between October 27 and November 9, ecommercefastlane.com took roughly 57,000 requests from OpenAI, Anthropic and Perplexity crawlers and agents, against about 936,000 from Bingbot and Googlebot. That is about six AI requests for every hundred from the two search engines. The number that changed my thinking was a different one: our site turned away 84 percent of PerplexityBot’s requests and 71 percent of Claude-SearchBot’s, while letting most of OpenAI’s search crawler and Anthropic’s training crawler through. This is an update to the version of this article we published in September. That one counted crawler visits a different way, so its figures are retired rather than refreshed, and it made two claims this window disproved: that no AI vendor crawler asked for our sitemap, and that nothing fetched an llms.txt. Both are corrected below. Last updated November 2026. One caveat before the numbers, and it matters. This site is a publisher, not a store, and we matched crawlers by their user-agent strings on sampled edge logs. User-agent strings can be faked, so I round the figures and you should treat them as directional. If you run a store, the transferable part is the method: look at what your own site did when the crawlers showed up.
Generative engine optimization is the work of making your store reachable, indexable and quotable inside AI-generated answers, and for a Shopify merchant I’d argue it is mostly access, index health and complete product data rather than a new content discipline. The term comes from a 2023 research paper by Pranjal Aggarwal and colleagues, the study that introduced generative engine optimization and the GEO-bench benchmark, which found that specific content changes could lift visibility in generative responses by up to 40 percent. Notice what that paper tested: changes to written passages, such as adding citations, quotations and statistics. That is a publisher’s playbook. A product page for a 14-ounce merino base layer in four colorways is a structured record with a price, an availability state, a size chart, a return window and a shipping promise, and an assistant deciding whether to mention it reads that record before it reads your prose. The field-level version of that work lives in our guide to the purchase decision facts most Shopify product pages leave out. This article is the operating model around it. Google’s own guide to optimizing for generative AI features in Search draws the same line from the other side: a page has to be indexed and eligible to show a snippet before it can appear in those features at all. So I break the work into three checks, and the rest of this piece follows them in order. Can the AI crawlers reach your pages? Are Bing and Google spending their visits on pages that still exist? Is the product data they find complete?
The agents of the three AI vendors we looked at, OpenAI, Anthropic and Perplexity, made about six requests for every hundred that Bingbot and Googlebot made on our site between October 27 and November 9, and all three were active. Anthropic’s agents made about 43,000 requests, OpenAI’s about 9,000 and Perplexity’s about 5,000, against roughly 936,000 from Bing and Google combined. The busiest AI visitor was Claude-User, at about 28,800 requests, which is the agent that fetches a page when a person asks Claude about it. Other crawlers dwarfed the assistants. ByteDance’s Bytespider alone made about 81,000 requests, more than the three AI vendors combined, and Applebot made about 54,000. None of that is evidence about what any assistant will say about you. It does tell you the traffic is real, it is not small, and most of it is invisible unless you look at your logs. Do not set these figures beside the ones in the September version. The counting method changed, so the old numbers are retired. What carries over is the direction. Bing and Google still dwarf the AI vendors, which is why index health stays one of the three checks, and the AI agents are present and fetching, which is why access is another.
Each of the three vendors runs separate agents for training, search and user-requested fetches, and robots.txt treats them differently, so blocking “OpenAI” or “Anthropic” as one thing blocks the wrong job. OpenAI’s crawler documentation says OAI-SearchBot surfaces sites in ChatGPT’s search features and follows robots.txt, GPTBot gathers content that may be used to train its models and follows robots.txt, and ChatGPT-User visits pages when a person asks, in which case robots.txt rules may not apply. Anthropic’s explanation of its crawlers pairs ClaudeBot for training with Claude-SearchBot for search quality and Claude-User for fetches on a user’s question, all controlled through robots.txt. Perplexity’s crawler guide says PerplexityBot surfaces sites in its search results and is not used for foundation model training, while Perplexity-User is a user-triggered fetch that generally ignores robots.txt. Now, the decision this sets up. Blocking a training crawler is a legitimate choice, and some publishers make it. Blocking a search crawler is a different choice: you are asking not to appear in that product’s results. Refusing a user-triggered fetch is different again, because a real person asked an assistant about your page and the assistant got an error. I’d argue most stores should make those three calls separately, on purpose, and write them down. On Shopify the place you make them is robots.txt, which you can customize through a robots.txt.liquid template, as Shopify’s documentation explains.
Our site refused most of what PerplexityBot, Claude-SearchBot and Perplexity-User asked for, while OpenAI’s search crawler and Anthropic’s training and user-fetch agents mostly got through. The split below covers October 27 to November 9, where “refused” means a 403, 429 or 503 response, and every figure is rounded.
Those codes come from a firewall rule, a rate limit or an overloaded server, and the logs do not say which. I lean toward rules over load, because a straining server would be unlikely to refuse PerplexityBot at 84 percent while refusing ClaudeBot at 3 percent, but I cannot prove that from these logs, and a rate limit would also catch a crawler that sends bursts. Here is what I took from it. A raw count of 4,400 PerplexityBot requests would have told me Perplexity takes real interest in the site. The status split says about 3,700 of them hit a wall, which means any visibility we lack in Perplexity may be partly something we did to ourselves. GPTBot’s 55 percent refusal is a different story, because that is a training crawler and refusing it can be policy. The search crawlers and the fetch-when-asked agents are the ones I would never refuse by accident. On November 10 we loosened the restrictions as a test, so any log window after that date will look different, and read later numbers with that in mind. The thing I would remove from your plan first is a visibility tracker bought before you have read your own refusal codes, because a tracker would have scored us low on Perplexity without telling us why. Tanner Larson of Built to Scale argued on episode 407 of the podcast that the large majority of Shopify stores, somewhere between 80 and 90 percent by his count, are invisible to AI platforms. I cannot verify that figure and I treat it as one guest’s estimate. What I can say is that one ordinary cause of invisibility sits in logs most merchants never open. A standard Shopify store generally does not let you read raw request logs, so your equivalents are your robots.txt and any security app, CDN or proxy sitting in front of the store. Bing Webmaster Tools and Google Search Console show how Bing and Google crawl you, which covers the index-health check, but they do not report on the AI crawlers. If you run Cloudflare or a headless storefront, count the 403, 429 and 503 responses by user agent and compare them with what you meant to allow.
Bingbot spent 52 percent of its 578,520 requests on URLs that returned a 410 Gone status, and Googlebot spent 36 percent of its 357,227, so deleting a page does not stop crawlers from asking for it. Only 23 percent of Bingbot’s requests and 44 percent of Googlebot’s returned a normal 200 response. Another 13 percent of Bingbot’s went to 404 errors, and about 17 percent of Googlebot’s went to 301 and 302 redirects. Those 410s are URLs we removed on purpose and told crawlers about. They kept asking. Why does that matter for AI answers? Google’s guide ties its generative features to being indexed and eligible for a snippet. The other assistants do not publish their sources, so I treat the Bing and Google indexes as the safest proxy for whether your pages can be found, not as a proven pipeline. Either way, a crawler spending half its visits on dead URLs may be spending visits that could have gone to live ones, though I cannot show from these logs that it costs us any rankings. The store version of this is dead products, retired collections and parameter URLs from filters. Shopify builds your sitemap for you, so the work is in the redirects. Where a discontinued product has a close replacement or a fitting collection, a 301 to it makes sense. Where it does not, an honest 404 or 410 is fine. Either way, stop linking internally to URLs that no longer exist. The same data corrected us on sitemaps. September’s version said no AI vendor crawler asked for ours. In this window Anthropic’s agents requested our sitemap files about 550 times, OpenAI’s about 60 and Perplexity’s about 10, and they requested robots.txt about 1,170, 250 and 165 times. Those requests show the vendors checked the rules and the map. I cannot tell from the logs what they did with them. What they find on your pages afterward is the part you control.
Shopify’s Fall ’26 Edition lists a set of tools aimed at AI channels, and nearly all of them are only as good as the product data and policies already in your admin. The edition page names a catalog eligibility check, a business details knowledge base for AI channels, compare-at prices that agents can read, product data mapping by channel, buy and ask endpoints on the Universal Commerce Protocol, WebMCP tools, a Shop skill for any agent, and selling through Google Search and Gemini. The page shows no release dates, so confirm what is live in your own admin before you plan around any item. The groundwork came earlier. On its Spring 2026 edition page, Shopify says it gets your products into AI channels automatically and claims that data syndicated by Shopify drives twice the conversion in AI chats. The page does not say who is eligible or what the comparison baseline is, so treat the 2x as Shopify’s number and verify your own setup. On episode 453 I gave three actions for this moment: confirm Agentic Storefronts is set up the way you expect, rewrite the titles and descriptions on your top 20 products, and track AI-referred attribution. All three still hold. The guest on episode 480 frames the naming half of that bluntly: a beautiful product name tells an AI very little about what the product is. If you sell outside Shopify, or want to see what good data looks like from the buyer side, OpenAI’s product feed specification documents the canonical fields. For agents that transact rather than recommend, see our complete guide to agentic commerce for Shopify merchants, and for the data work itself, our guide to Shopify product data enrichment.
AI-referred visitors are worth counting properly even when there are few of them, because Adobe’s September 2026 retail data shows them converting far better than other traffic. Adobe’s analysis published on October 29 found AI traffic to US retail sites up 71 percent year over year, converting 90 percent better than other traffic, producing 83 percent higher revenue per visit and holding visitors on site 73 percent longer. Those are aggregates across US retail, not a promise about your store. They replace the March figures the September version of this article used. Our own number is smaller and messier. Our analytics for October 27 to November 9 show roughly 400 AI-referred sessions spread across 207 landing pages. The homepage took 42, a 2023 article about international warehouses took 30, and 27 had no landing page recorded. Our pillar on this very topic received none, and our rank data shows it ranking for nothing yet. I read the spread as the lesson. Assistants send people to pages we never planned for, which argues for completeness across the whole site over polishing one hero page. You can measure this without buying a dashboard. Segment referral sessions by assistant domain, keep a log of twenty buyer questions that you run monthly in two assistants, and follow the routine in our 30-minute manual check for AI search visibility on Shopify. For how stores compare across assistants at larger scale, see our write-up of the StoreHero AI search visibility benchmark.
An llms.txt file costs an afternoon, Google says it is unnecessary for its generative features, and we have no evidence it drives citations, so write one only if you will keep it accurate and do not pay anyone to build it. Google’s AI optimization guide says you do not need llms.txt or other special markup, and its list of things to skip includes breaking content into tiny chunks and rewriting content for AI systems. It also says structured data is not required for its generative features. Keep the schema your theme already outputs, because it still helps classic search. The warning is against adding six schema types to a page that still does not state its return window in plain text. The correction: September’s version said nothing had fetched an llms.txt on our domain. In this window there were requests for /llms.txt and for our Key Facts page for AI systems, but most came from generic browser agents and audit tools, many were refused, and the one clearly AI-vendor hit was a ChatGPT user agent on that Key Facts page. That hit was on the Key Facts page, not on the llms.txt file itself, and one hit is not proof either matters. It is enough to retire the claim that nothing fetches them. My position moved from skip it to cheap and optional. We now publish both, because a short, curated page of facts about the business costs little and at least one assistant agent has fetched it. If a vendor charges you for one, ask for fetch logs showing which AI agents read it. We covered Google’s wider statements in our fact check of what Google said about AI search, and the conclusion has held.
The right first move depends on revenue, because the constraint at $50K is completeness and the constraint at $10M is attribution. Doing the $2M work at $50K is the premature complexity I see stall stores in that band. The revenue cutoffs in the table are my judgment, not tested thresholds.
A note on the middle rows, because that is where most readers of this site sit. Between $500K and $2M the temptation is to hire the problem away, and I’d argue against it until the first rows are clean, because much of what an assistant needs to recommend you lives in your product data and policies. An agency cannot write your return window for you. What an agency can do is the corroboration and measurement work in the $2M row, which is why hiring one before the foundation is clean is money spent to hide a foundation problem. When you are ready, start with our guide to hiring a digital marketing agency for a Shopify store.
Three kinds of spend in this space are not worth it for most merchants right now, by my rule of thumb. The first is AI visibility monitoring software below roughly $2M in revenue. The dashboards are real, but a visibility score does not tell you what to fix, and at your size the twenty-question spreadsheet gives you the same signal for an hour a month. The second is anything sold as an llms.txt installation or AI-specific markup service, since the file is cheap to write yourself and Google calls the markup unnecessary. The third is AI-targeted rewrites of content that already works, which is the most expensive of the three because it consumes months and aims at a problem your product records, not your prose, are more likely to have. Before any of those, read what your own site did when the crawlers arrived. It is free. The pattern I keep seeing at $500K to $2M is premature complexity: too many apps, too many channels and too many tactics stacked on fundamentals that are not solid yet. Apply the eighteen month test to anything in this category. Clean product data will still matter in eighteen months, and so will an explicit return policy and being indexed. A dashboard score that tells you where you rank without telling you what to fix may not. If you are weighing outside help anyway, our field guide to AI SEO agencies for Shopify and DTC brands lays out which ones fit which stage and takes the same position: fix fundamentals first.
Not much, because the same fundamentals decide whether an AI answer can find your pages. Google’s guidance says a page must be indexed and eligible to appear with a snippet before it can show up in its generative AI features, which makes index health a prerequisite rather than a separate project. What changes for a store is emphasis. Classic ecommerce SEO leans on keywords, titles and links. Generative engine optimization leans harder on complete product data, explicit policies, crawler access and corroboration from sources you do not own. The infrastructure is shared, and only the pressure points differ.
Decide by job, not by vendor. Blocking a training crawler such as GPTBot or ClaudeBot is a policy choice many publishers make. Blocking a search crawler such as OAI-SearchBot, PerplexityBot or Claude-SearchBot asks not to appear in that product’s results, so do it only on purpose. The fetch-when-asked agents, ChatGPT-User, Claude-User and Perplexity-User, run when a real person asks about your page, and the vendors say some of them may ignore robots.txt. Write your three decisions down, then check that your site’s actual behavior matches them instead of assuming a setting or app is doing what you meant.
No, but it is cheap and optional. Google says creating llms.txt files and other special markup is unnecessary for its generative AI features, and we have no evidence that a file drives citations. In our own logs the one clearly AI-vendor request was a ChatGPT user agent fetching our Key Facts page, a related page rather than the llms.txt file itself, so the old claim that nothing fetches these files no longer holds. If you write one, keep it short and accurate: shipping, returns, contact details and your best collections. Do not pay for a tool or a service to build it, and put indexing and product data ahead of it.
Check three places: your robots.txt, anything sitting in front of your store such as a security app, CDN or proxy, and the response codes your logs show for each AI user agent. A standard Shopify store generally does not expose raw logs, so start with robots.txt and any bot-protection app. Bing Webmaster Tools and Google Search Console show how Bing and Google crawl you, not the AI crawlers, so use them for index health only. If you run Cloudflare or a headless storefront, count the 403, 429 and 503 responses by user agent. A bot that gets mostly refusals is being blocked, whether you meant to block it or not.
Possibly, but verify it rather than assume it. Shopify’s Spring 2026 edition page says it gets your products into AI channels automatically, without saying who is eligible, and the Fall ’26 edition lists more AI channel tools without release dates. The place to check is Agentic Storefronts in your Shopify admin, which shows which channels are live for your store. Shopify claims data syndicated this way drives twice the conversion in AI chats, which is the company’s own figure. Check the screen before you buy anything, because it shows what is live and what your product data is missing.
Fill in every required product field, publish shipping and returns policies in plain text, and confirm your product pages are indexed in both Google and Bing. That is the core list, and it beats anything else available to you at that size. Then check your robots.txt and any bot-protection app so you know you are not turning AI crawlers away by accident, and open Agentic Storefronts to see which channels are live. After that, make sure your twenty best sellers answer the questions a buyer asks before purchase, including sizing, materials, delivery time and how returns work.