Skip to content
Orino Docs

OAI-SearchBot blocked in robots.txt

What this means

Section titled “What this means”

Your robots.txt blocks OAI-SearchBot with Disallow: /. OAI-SearchBot is ChatGPT’s real-time web search crawler — the bot that powers citations when a user asks ChatGPT something with web search enabled. Blocking it means your site cannot appear in those answers, regardless of how relevant your content is.

This is distinct from GPTBot, which crawls for training data. You can block GPTBot (to opt out of training) while keeping OAI-SearchBot allowed for real-time citations. Blocking OAI-SearchBot removes you from live ChatGPT results entirely.

How to fix it

Section titled “How to fix it”

Remove the OAI-SearchBot block from your robots.txt:

# Remove these lines
User-agent: OAI-SearchBot
Disallow: /

To allow OAI-SearchBot site-wide, delete both lines. To exclude only specific paths:

User-agent: OAI-SearchBot
Disallow: /internal/
Disallow: /staging/

Verify the fix

Section titled “Verify the fix”

After deploying:

Terminal window
curl -s https://yourdomain.com/robots.txt | grep -A 3 "OAI-SearchBot"

No result means OAI-SearchBot falls back to your wildcard rules. Disallow: / means it is still blocked. Re-run orino audit to confirm the check passes.

Section titled “Related fixes”