Multi-bot robots.txt Tester
Test your robots.txt against 18 user-agents, classic search engines and AI bots, with RFC 9309 compliant rule priority. You decide who can crawl your site.
Your robots.txt
We append /robots.txt automatically if absent.
Access matrix
| Bot | Family | Paths |
|---|
View the analyzed robots.txt content
Also worth exploring
All toolsVerify your hreflang tags: valid ISO codes, reciprocity between language versions, x-default, conformance with Google's recommendations.
Inspect HTTP response headers: x-robots-tag, cache-control, HSTS, link, vary, server. SEO + security annotations on impactful headers.
Validate your sitemap.xml: structure, URL count, lastmod outliers, sitemap-index recursion. Conforms to sitemaps.org and Google.
Test up to 200 URLs at once: HTTP code, redirect chain, response time. Ideal for backlink profile audits.
Frequently asked questions
Which user-agents are tested? +
This tool tests 18 crawlers across 4 families. Search engines (Googlebot, Bingbot, DuckDuckBot, YandexBot, Baiduspider), image / specialty (Googlebot-Image, Googlebot-News), generative AI (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, anthropic-ai, PerplexityBot, Perplexity-User, Applebot, Applebot-Extended, Google-Extended, CCBot), and the * wildcard. The AI family evolves quickly, we adjust regularly.
How does rule priority work? +
According to the Robots Exclusion protocol (RFC 9309 since 2022), the longest rule matching the path wins, regardless of order in the file. If Allow: /blog/ and Disallow: /blog/draft/ are defined, /blog/draft/2024.html is blocked because the Disallow rule is more specific. On exact length tie, Allow wins at Google.
Why test against AI bots? +
Blocking GPTBot, ClaudeBot, PerplexityBot prevents your content from training these models or being cited in their answers. Conversely, allowing these bots helps your brand appear in ChatGPT, Claude, Perplexity. It's a strategic trade-off (AI visibility vs intellectual property) that deserves an explicit decision, not an inherited default.
My robots.txt is not publicly accessible, can I still test it? +
Yes. Paste the content directly into the second "Paste content" tab instead of providing a URL. The test runs locally in the browser, with no transmission to a third-party server.
Beyond robots.txt, who is talking about your site in LLMs?
Stringer helps brands position themselves in content reused by ChatGPT, Claude and Perplexity through links and mentions on French media.