For almost every small and local business, the answer is: let them crawl. Being findable and citable by AI systems is worth more to you than the content you’d be protecting, and the businesses opting out are mostly doing it on a reflex rather than a calculation.
That said, the controls are real now and worth understanding. Google began honouring a Search Console toggle on June 17, 2026 that removes your site from appearing in and grounding generative AI features. robots.txt directives for AI user agents are widely respected by the major crawlers. And llms.txt, the file the industry spent eighteen months talking about, turns out to be barely fetched by anyone.
Here’s what each control actually does, who should use it, and where the honest uncertainty is.
Three Different Questions People Keep Conflating
Most of the confusion in this space comes from treating one decision as three, or three as one. They’re separate, and you can answer them differently.
Can AI systems use my content to answer questions and cite me? This is search-time retrieval - AI Overviews, AI Mode, ChatGPT with browsing, Perplexity. Blocking it removes you from those answers.
Can AI companies use my content to train models? This is training-time ingestion. Blocking it has no effect on whether you appear in AI search results, because those run on live retrieval.
Can AI agents acting for a specific user read my site? This is an agent doing a job for a person who wants to buy from or book with you. Blocking it is almost always self-defeating, and it’s covered in can an AI agent actually buy from your website.
Someone who objects to their work training a commercial model does not necessarily want to disappear from AI search results. Those are different levers, and conflating them is how sites end up invisible by accident.
What Each Control Actually Does
robots.txt for AI user agents
The oldest and most broadly respected mechanism. You can name specific AI crawlers and disallow them. The major ones - GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot, Google-Extended, CCBot - do generally honour it.
Two things people get wrong. First, the distinction between crawlers: some fetch for training, some for live search-time retrieval, and blocking the wrong one produces the opposite of what you intended. Google-Extended, for instance, is about training and grounding rather than ordinary Search indexing. Second, robots.txt is a request, not a wall. Well-behaved crawlers respect it; anything that doesn’t want to won’t. If you need enforcement rather than a polite notice, that’s a server-level job.
Google’s Search Console AI toggle
New in 2026, honoured since June 17. It lets you decide whether your site appears in and helps ground generative AI features including AI Overviews, AI Mode, and AI Overviews in Discover.
This is the highest-stakes switch on the list, because it operates on the surface where a large share of searches now end. For a local service business trying to be the name an AI recommends, turning it off is usually a straightforward loss. It sits next to the new AI performance reports, which we cover in seeing whether Google’s AI is showing your site - and looking at your own data for a few months before flipping anything is the sane order of operations.
Content Signals in robots.txt
A newer convention that lets you express intent per use case rather than blanket-blocking a crawler - separating search indexing from AI training, for example. It’s a more precise instrument than an all-or-nothing disallow, and it’s the approach we use on this site: allow the retrieval that makes us findable, express a position on training separately.
Adoption is still uneven. Treat it as a clear declaration of intent that some crawlers honour, not as an enforcement mechanism.
llms.txt
This one deserves a plain assessment, because it has been oversold.
llms.txt is a proposed file listing your site’s key content in a machine-friendly format. The pitch was that AI systems would read it and understand your site better. The data hasn’t supported the pitch. Adoption sits around ten percent of domains after eighteen months of discussion, and a large share of those files are empty plugin stubs. More tellingly, the crawlers mostly ignore it: one measurement of 62,100 AI bot visits over ninety days found only 84 requests for the file, about 0.1 percent of AI crawler traffic. Google’s Gary Illyes stated in 2025 that Google doesn’t support llms.txt and isn’t planning to. Microsoft and OpenAI crawlers do fetch it in some cases.
So the honest position: llms.txt is not an SEO tactic and will not improve your AI search visibility today. It costs about an hour, it forces you to write down a clear map of your own information architecture, and it might matter more later if agent-facing standards consolidate. Do it if you want a cheap hedge. Don’t do it expecting a ranking effect, and be wary of anyone selling it as one.
Who Should Actually Block
There are legitimate cases. They’re just narrower than the discourse suggests.
Publishers and content businesses whose content is the product. If people pay for your writing, research, or data, giving it away to a system that answers the question without sending anyone to you is a genuine business problem. Several large publishers have made this call deliberately, some with licensing deals attached. This is a real decision with real trade-offs, not a reflex.
Sites with genuinely proprietary material. Original research, paid databases, licensed datasets. Block those paths specifically. You don’t need to block the whole site to protect one directory.
Anyone with a licensing arrangement where exclusivity is part of the deal.
Anything behind a login should be inaccessible anyway, and if it isn’t, that’s an access control problem rather than an AI problem.
For everyone else - local services, trades, clinics, restaurants, agencies, most B2B, most e-commerce - the calculation runs the other way. Your content isn’t the product; it’s marketing for the product. Being the business an AI names when someone asks for a recommendation is worth considerably more than the marginal value of the words. Blocking is choosing to be absent from the conversation where buyers now start.
What We Actually Recommend
For a typical small or local business:
Allow AI search and retrieval crawlers. This is how you get cited, recommended, and found. Leave the Search Console AI toggle alone.
Decide on training separately, and express it precisely. Use Content Signals or a targeted robots.txt directive if you have a view on model training. It costs you nothing in search visibility because those are different pipelines.
Block specific paths, not the whole site, if you have genuinely proprietary content. Precision beats a blanket rule.
Skip llms.txt unless you want the hedge. If you do add one, keep it accurate and don’t expect it to do anything measurable this year.
Then spend your energy on being worth citing. Which is the real point. Crawler permissions determine whether you’re eligible. Content quality, structure, and reputation determine whether you’re chosen. Most businesses agonising over robots.txt have a content problem, not an access problem. Our Generative Engine Optimization guide for Canada covers the part that actually moves the needle.
TheBomb® has been building and maintaining sites since 2014, and we’ve taken these decisions on our own site as well as clients’. Our position is the one above: open for retrieval, deliberate about training, and far more focused on being worth quoting than on gatekeeping.
Frequently Asked Questions
Should I block AI crawlers on my website?
For most small and local businesses, no. Your content is marketing for what you sell rather than the product itself, and being cited or recommended by AI systems is worth more than the content you would be withholding. Blocking makes sense mainly for publishers whose writing is the product, sites with genuinely proprietary research or data, and anyone under a licensing arrangement requiring exclusivity.
Does blocking AI training also remove me from AI search results?
Not necessarily, because they run through different pipelines. Training-time ingestion is about building models, while AI Overviews, AI Mode, and similar features rely on live search-time retrieval. You can express a position on training while remaining fully available for retrieval, though you need to target the correct user agents to achieve that, since blocking the wrong one can remove you from search features unintentionally.
Does llms.txt actually work?
Not in the way it has been marketed. Adoption sits around ten percent of domains, many of those files are empty plugin stubs, and measurement shows AI crawlers fetch it very rarely - one study found llms.txt accounted for roughly 0.1 percent of AI bot requests. Google has stated it does not support the file and has no plans to. It costs about an hour and may matter more later, but it will not improve your AI search visibility today.
What is the Google Search Console AI toggle?
It is a control added in 2026 that lets site owners decide whether their content appears in and helps ground Google’s generative AI features, including AI Overviews, AI Mode, and AI Overviews in Discover. Google began honouring the setting on June 17, 2026. Because it affects the surface where a growing share of searches now end, turning it off usually costs a local business meaningful visibility, so it is worth reviewing your own AI performance data before changing it.
Which AI crawlers should I name in robots.txt?
The commonly named agents are GPTBot and OAI-SearchBot from OpenAI, ClaudeBot from Anthropic, PerplexityBot, Google-Extended, and CCBot from Common Crawl. They serve different purposes, with some fetching for training and others for live retrieval, so identify what each one does before disallowing it. Note also that robots.txt is a request rather than enforcement, so anything requiring a hard block needs handling at the server level.
Not Sure What Your Site Is Currently Telling Crawlers?
Plenty of sites are blocking things they don’t mean to, usually from a decision made in 2024 and never revisited. If you want to know what your robots.txt and settings are actually saying, get in touch and we’ll audit it. See our development and SEO services for how we handle the technical foundation.
Key Takeaways
- Most small businesses should let AI crawl. Your content is marketing, not the product, and being cited is worth more than withholding it.
- Three separate questions: AI search retrieval, model training, and user-directed agents. You can answer them differently.
- Google’s AI toggle has been honoured since June 17, 2026 and removes you from AI Overviews and AI Mode. High stakes, so review your data first.
- llms.txt is not an SEO tactic. Around 10% adoption, roughly 0.1% of AI bot requests, and Google has said it doesn’t support it.
- Block paths, not sites, if you have genuinely proprietary content. Precision beats a blanket disallow.
- robots.txt is a request, not a wall. Well-behaved crawlers honour it; enforcement is a server-level job.
- Access determines eligibility; quality determines selection. Most businesses worrying about robots.txt have a content problem instead.
