Skip to content
Orino Docs

robots.txt missing

What this means

Section titled “What this means”

Every site should have a robots.txt file at /robots.txt. Without one, crawlers have no guidance about which paths to index and which to skip. Most crawlers will index everything by default, but you lose the ability to protect internal routes, reduce crawl budget waste on low-value pages, and point crawlers to your sitemap.

How to fix it

Section titled “How to fix it”

Create a robots.txt in the location your framework serves static files from.

Next.js App Router

Section titled “Next.js App Router”

The recommended approach is app/robots.ts, which Next.js compiles to /robots.txt automatically at build time:

app/robots.ts
import type { MetadataRoute } from 'next'


export default function robots(): MetadataRoute.Robots {
  return {
    rules: [
      {
        userAgent: '*',
        allow: '/',
        disallow: ['/api/'],
      },
    ],
    sitemap: 'https://yourdomain.com/sitemap.xml',
  }
}

Alternatively, place a static public/robots.txt in your project root.

Next.js Pages Router

Section titled “Next.js Pages Router”

Create public/robots.txt:

User-agent: *
Allow: /


Sitemap: https://yourdomain.com/sitemap.xml

SvelteKit

Section titled “SvelteKit”

Create static/robots.txt:

User-agent: *
Allow: /


Sitemap: https://yourdomain.com/sitemap.xml

Nuxt

Section titled “Nuxt”

Create public/robots.txt:

User-agent: *
Allow: /


Sitemap: https://yourdomain.com/sitemap.xml

For programmatic control over multiple environments, use the @nuxtjs/robots module.

Astro

Section titled “Astro”

Create public/robots.txt:

User-agent: *
Allow: /


Sitemap: https://yourdomain.com/sitemap.xml

Plain HTML

Section titled “Plain HTML”

Place robots.txt in the root directory of your site, alongside index.html:

User-agent: *
Allow: /


Sitemap: https://yourdomain.com/sitemap.xml

Verify the fix

Section titled “Verify the fix”

Check the file is accessible at its standard path:

Terminal window
curl -I https://yourdomain.com/robots.txt

The response should return 200 OK. Check the content is correct:

Terminal window
curl https://yourdomain.com/robots.txt

Re-run orino audit to confirm the check passes.

Section titled “Related fixes”