Skip to content
Orino Docs

GPTBot blocked in robots.txt

What this means

Section titled “What this means”

Your robots.txt has Disallow: / under User-agent: GPTBot. This tells OpenAI’s crawler to leave your site entirely, which keeps your content out of OpenAI’s training corpus and knowledge systems.

How to fix it

Section titled “How to fix it”

Find and remove the GPTBot block in your robots.txt:

# Remove these lines
User-agent: GPTBot
Disallow: /

If you want to allow GPTBot site-wide, delete both lines. If you want to exclude only specific paths, use a targeted disallow:

User-agent: GPTBot
Disallow: /private/
Disallow: /members-only/

Verify the fix

Section titled “Verify the fix”

After deploying, run:

Terminal window
curl -s https://yourdomain.com/robots.txt | grep -A 3 "GPTBot"

If the command returns nothing, GPTBot has no specific rule and falls back to your wildcard rules. If it returns Disallow: /, the block is still active. Re-run orino audit to confirm the check passes.

Section titled “Related fixes”