Which AI crawlers should I allow in my online store’s robots.txt?

No. OpenAI distinguishes its search crawler, OAI-SearchBot, from GPTBot, whose requests can support model training. These settings are independent. Declining training access therefore does not automatically require blocking the search crawler. The current OpenAI crawler documentation is the reference.

Allowing a crawler does not force a recommendation of your store. It may remove an access barrier. Which sources the system subsequently selects is a separate decision. For merchants, this distinction matters: technical access and a successful GEO outcome are not the same thing.

The purpose of each request matters at Google too. Googlebot controls are relevant to Search features, including AI Overviews and AI Mode. Google-Extended concerns certain uses in other Google AI products; it is not a switch that exclusively enables or disables AI answers within Google Search.

Google explains this distinction in its documentation on AI features in Search and its crawler overview. Before changing anything, we would record which service should be allowed to access which public content, and what the rule is intended to achieve.

Robots.txt is only one layer. A CDN, firewall or bot protection service can still reject a request or demand a browser check. Conversely, opening a page successfully in your own browser does not prove a search crawler receives the same content.

We would start with an affected product or advice page and inspect response status, redirects and the content returned. An error page with HTTP 200 is no more useful than a CAPTCHA in front of the product description. In PrestaShop, hosting, upstream services and relevant security modules therefore belong in the same investigation.

Changing a request’s User-Agent name alone is not reliable proof of access by the actual provider. Server logs should be interpreted using the provider’s documented verification methods or published IP ranges. We would not disable bot protection wholesale; we would investigate a justified, narrowly scoped exception.

• Purpose: distinguish search visibility, training and user-initiated requests.
• Rules: inspect the robots.txt actually served by the affected host.
• Pages: select important categories, products and guides as concrete test cases.
• Access: compare HTTP responses, redirects, bot protection and visible content.
• Evidence: use logs and provider documentation to identify requests.
• Follow-up: document the change and observe the affected access again.

This produces an outcome that can be checked: which rule blocks which page, and what changed? That is more useful than a general label such as “AI-friendly store” which says nothing specific about accessibility.

No. It instructs cooperating crawlers; it is not access control. Customer accounts, internal documents and test stores need appropriate authentication or access restrictions. Listing confidential paths in robots.txt does not make them confidential.

Crawling and indexing also need to be distinguished. Google explains that a blocked URL can sometimes still appear in its index. Google needs to fetch a noindex instruction to process it. Its robots.txt introduction covers the principles. Our article on duplicate content discusses which store URLs should be indexed in the first place.

A llms.txt file does not replace these decisions. We cover that separately in Does an online store need llms.txt?. Another file achieves little if access to the actual page fails at a blocking layer.

Consider a hypothetical case: following a change in bot protection, a guide remains accessible to visitors while automated search requests encounter a challenge page. We would isolate the change, investigate the affected requests and coordinate a controlled correction with the hosting provider. This describes an assessment process, not a documented customer case.

Our GEO consulting can be connected with technical PrestaShop support here. PrestaShop provides scope for targeted changes; replacing the store platform is normally not the first response to an incorrect access rule. After the technical assessment comes the content question: is the accessible information useful and clear? Whether visibility improves is a separate investigation, using a measurement plan for AI visibility.