Relaunch: AI visibility and crawler exposure audit. Full fix package is $19.
Anthropic crawler policy

Claude SearchBot Robots.txt Guide

Robots.txt patterns for Claude-SearchBot, ClaudeBot and Claude-User, with a practical checklist for search visibility and training opt-outs.

When to use this

Use this when you want Claude search visibility without accidentally treating every Anthropic crawler as the same bot.

Separate Claude bot jobs

Anthropic documents distinct user agents. Claude-SearchBot is used to improve search result quality, ClaudeBot is used for web crawling, and Claude-User is tied to user-initiated requests.

  • Keep Claude-SearchBot available when search visibility is the goal.
  • Use ClaudeBot restrictions when your policy is to limit training or broad crawling.
  • Review Claude-User separately if user-directed retrieval matters for your product.
User-agent: Claude-SearchBot
Allow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-User
Allow: /

Sitemap: https://example.com/sitemap.xml

Avoid false positives

A robots.txt rule only describes crawler permission. CDN bot fights, login walls, broken canonical tags and thin initial HTML can still make the page unusable as a source.

Checklist

Confirm Claude-SearchBot is not blocked by a wildcard Disallow rule.

Check that the robots.txt file is reachable on every subdomain you care about.

Avoid relying on IP blocks as the only opt-out mechanism because crawlers need to read robots.txt.

Test important URLs for initial HTML text, headings and canonical consistency.

FAQ

What does Claude SearchBot Robots.txt Guide check first?

Confirm Claude-SearchBot is not blocked by a wildcard Disallow rule.

Does this guarantee ranking or inclusion in AI answers?

No. It checks public technical signals that can make a page easier to crawl, parse and cite, but no tool can guarantee ranking, indexing or citation in ChatGPT, Claude, Perplexity or Google.

Should I fix robots.txt, llms.txt or page rendering first?

Fix public reachability, indexability and readable initial HTML first. robots.txt should express crawler policy, and llms.txt is optional supporting documentation rather than a replacement for normal search fundamentals.

Primary references

Next step

Scan your site to see whether Claude-style crawler access and source readability are aligned.

Scan your site