LLM Discovery
Robots.txt and AI Bots
The robots.txt file tells compliant web crawlers which parts of a website they should or should not crawl. That includes traditional search engine bots and, in many cases, AI crawlers operated by companies developing large language models.
Over the past few years, robots.txt has gone from being a technical SEO file that almost nobody thought about to something businesses suddenly want to debate over coffee.
Should you block AI?
Should you allow AI?
Should you block every bot except Google?
The answer, as with most things in SEO and AI, is: it depends.
Robots.txt controls crawling, not whether your business exists on the internet. Blocking a crawler does not erase information that already exists elsewhere.
What Is robots.txt?
The robots.txt file lives at the root of a website, typically at:
https://example.com/robots.txt
It contains instructions for automated crawlers using simple directives such as allowing or disallowing access to certain paths.
For example:
User-agent: *
Disallow: /private/
This tells compliant crawlers not to crawl the /private/ directory.
Does robots.txt Block AI?
Sometimes.
Many AI companies publish crawler user-agent names that respect robots.txt directives.
If you block one of those user agents, its crawler may stop accessing the sections you've restricted.
However, robots.txt is not a universal AI off switch.
Different companies use different crawlers, different retrieval systems, and different data sources.
Some AI-generated answers may rely on information retrieved from search engines, licensed datasets, public business profiles, or third-party websites rather than your own pages.
Why Some Businesses Block AI Crawlers
- They publish paid or proprietary content.
- They want tighter control over how content is accessed.
- They have licensing concerns.
- They are protecting intellectual property.
- They simply prefer not to participate.
Those are business decisions.
There isn't one correct answer for everyone.
Why Some Businesses Allow AI Crawlers
Many businesses want greater visibility in AI-powered search and discovery.
Allowing compliant AI crawlers to access publicly available documentation may increase the amount of information AI systems can evaluate over time.
If your goal is to be discovered, understood, and recommended, blocking every AI crawler may work against that objective.
What robots.txt Does Not Do
| robots.txt Can | robots.txt Cannot |
|---|---|
| Provide crawl instructions. | Delete existing information from AI systems. |
| Help organize crawler access. | Guarantee privacy. |
| Restrict compliant bots. | Force every crawler to obey. |
| Reduce unnecessary crawling. | Improve AI visibility by itself. |
AI Visibility Is Bigger Than robots.txt
Let's be honest.
Some people act as though changing one line in robots.txt will determine whether AI recommends their business.
It won't.
Robots.txt is a technical configuration file.
It is not a visibility strategy.
AI systems evaluate businesses using documentation, structured data, reviews, citations, authority, entity recognition, internal linking, and dozens of other signals.
Those signals matter far more than a single crawl directive.
Best Practices
- Know why you're blocking or allowing a crawler.
- Don't copy someone else's robots.txt without understanding it.
- Review your robots.txt after website redesigns.
- Avoid accidentally blocking important public pages.
- Keep XML sitemap references up to date.
- Test your robots.txt regularly.
How Firm IQ Thinks About robots.txt
We see robots.txt as a policy document, not a marketing strategy.
If your business wants to participate in AI discovery, your website should generally be crawlable, technically healthy, and well documented.
If your business has legitimate reasons to restrict certain crawlers, that's a business decision that should be made deliberately—not because of internet rumors.
Most of the time, businesses have much larger opportunities waiting in their documentation than in their robots.txt file.
Robots.txt influences who can crawl your content. It does not replace strong documentation, technical SEO, or AI visibility fundamentals.