Technical SEO
Robots.txt
Robots.txt is a text file that gives crawl instructions to search engines and other automated bots.
It usually lives here:
https://example.com/robots.txt
That little file can quietly help or quietly wreck your visibility.
No drama. No flashing lights. Just one bad line telling crawlers not to look at the pages you actually need discovered.
Robots.txt controls crawling. It does not directly control indexing, rankings, or whether AI systems trust your business.
What Robots.txt Does
Robots.txt tells compliant crawlers which parts of a website they should or should not crawl.
For example, a site might allow crawlers to visit public pages but block private admin folders, duplicate search-result pages, or development areas.
A basic robots.txt file might look like this:
User-agent: *
Disallow: /admin/
Sitemap: https://example.com/sitemap.xml
That tells bots not to crawl the admin folder and points them toward the XML sitemap.
What Robots.txt Does Not Do
This is where people get into trouble.
Robots.txt is not a privacy tool.
It is not a noindex tag.
It is not a magic shield.
| Robots.txt Can | Robots.txt Cannot |
|---|---|
| Give crawl instructions. | Guarantee every bot will obey. |
| Reduce crawling of unimportant areas. | Remove a page from search results by itself. |
| Point crawlers to a sitemap. | Protect sensitive information. |
| Help manage crawl access. | Improve rankings by itself. |
Common Robots.txt Mistakes
- Blocking the entire website by accident.
- Blocking important service or location pages.
- Blocking CSS or image files needed to render pages properly.
- Using robots.txt instead of noindex.
- Forgetting to update the file after a redesign.
- Copying another website’s file without understanding it.
- Leaving development rules active after launch.
We see this all the time.
A business launches a new site and wonders why visibility disappears.
Then someone checks robots.txt and finds the digital equivalent of “please ignore everything.”
Robots.txt and XML Sitemaps
Robots.txt often includes a link to the site’s XML sitemap.
That helps crawlers discover important URLs more efficiently.
It does not replace internal linking, and it does not guarantee indexing.
But it is a good housekeeping habit.
Robots.txt and AI Bots
Robots.txt also matters for AI crawlers.
Some AI companies publish crawler names that website owners can allow or block through robots.txt.
Whether that makes sense depends on your goals.
If your goal is stronger AI visibility, blocking AI crawlers may reduce the amount of information they can access from your site.
This overlaps with Robots.txt and AI Bots in the LLM Discovery section.
Best Practices
- Keep the file simple.
- Block only what truly should not be crawled.
- Do not use robots.txt to hide sensitive information.
- Include your sitemap URL.
- Review the file after every redesign or migration.
- Test important pages to make sure they are crawlable.
- Coordinate robots.txt with indexing, canonicals, and redirects.
How Firm IQ Thinks About Robots.txt
Robots.txt is not strategy.
It is plumbing.
Important plumbing, sure. But still plumbing.
The goal is not to obsess over the file. The goal is to make sure it does not get in the way of discovery.
The truth is, as uncomfortable as it may be, a lot of technical SEO wins come from removing dumb obstacles.
Robots.txt is one of the first places to check.
A clean robots.txt file will not make weak content strong. But a bad robots.txt file can keep strong content from being discovered.