SEO Wiki
robots.txt
in short
robots.txt is a text file at the site root that controls search bot access to sections of a website.
robots.txt is a file with User-agent, Allow, Disallow and Sitemap directives read by search and AI crawlers.
How it works
A bot first fetches /robots.txt and follows its rules. Disallow blocks crawling but doesn't guarantee removal from the index.
Why it matters
- Hides service sections
- Points to sitemaps
- Lets you manage AI bots (GPTBot, ClaudeBot)
How Accendix handles it
We make sure robots.txt doesn't block CSS/JS or key pages and lists every sitemap.
Related pages
want this on your site?We'll break down your site in an SEO audit
Order SEO auditAll Wiki pagesUpdated: October 26, 2026