The Definitive Guide to llms.txt: Make Your Website AI-Ready

1. What is llms.txt?

In the early days of the web, robots.txt gave search engine crawlers a map. It told Google which pages to index and which to ignore. That simple text file shaped how billions of pages were discovered.

llms.txt does something similar — but for a fundamentally different kind of reader: large language models.

Proposed in 2024 by Jeremy Howard (creator of fast.ai and co-founder of Answer.AI), the llms.txt specification addresses a growing problem: LLMs are terrible at reading websites.

Modern web pages are soup — navigation bars, cookie banners, JavaScript widgets, ads, footers. An LLM trying to extract useful information from raw HTML wastes most of its limited context window on irrelevant markup. And converting HTML to clean text is unreliable at best.

llms.txt solves this by providing a single, curated Markdown file at /llms.txt that gives AI agents exactly what they need: a concise overview of your site, its structure, and links to deeper content — all in clean, LLM-friendly Markdown.

Think of it as the README for AI. If robots.txt says “what you can access,” llms.txt says “what you should understand.”

2. The Specification

The format is intentionally simple — it's Markdown with a specific structure that's both human-readable and machine-parseable. Here's the anatomy:

Required: H1 Title

Every llms.txt file must start with an H1 heading containing the project or site name. This is the only required element.

Optional: Blockquote Summary

A blockquote immediately after the H1 provides a short, high-signal summary. This is the elevator pitch — the context an LLM needs before reading anything else.

Optional: Body Text

Any Markdown content (paragraphs, lists, etc.) except headings can appear next. Use this for important notes, caveats, or context about how to interpret the links.

Optional: H2 Sections with File Lists

H2 headings delimit sections of links. Each link is a Markdown list item with a hyperlink, optionally followed by a colon and description.

Special: The “Optional” Section

An H2 titled exactly ## Optional has special meaning: links listed here can be skipped when a shorter context is needed. Use it for supplementary resources.

Here's the full structure:

# Your Site Name

> A one-line summary of what your site/product does.

Important notes or context about the site that LLMs
should know before diving into the links below.

## Docs

- [Getting Started](https://example.com/docs/start.md): Quick setup guide
- [API Reference](https://example.com/docs/api.md): Full API documentation

## Guides

- [Authentication](https://example.com/guides/auth.md): OAuth2 and API keys
- [Webhooks](https://example.com/guides/webhooks.md): Real-time event handling

## Optional

- [Changelog](https://example.com/changelog.md): Version history
- [Migration Guide](https://example.com/migrate.md): Upgrading from v1 to v2

Note that linked resources ideally point to .md files (or URLs with .md appended) so LLMs get clean Markdown rather than HTML. The spec also recommends that sites serve Markdown versions of pages at the same URL with .md appended.

3. llms.txt vs llms-full.txt

The original spec introduced a pattern used by the FastHTML project: expanding the link-based llms.txt into fuller context files. While the naming isn't formally standardized, a common convention has emerged:

When to use each:

• llms.txt — Use this as your default. It's small, fast to parse, and gives AI agents a map of your content. Agents that need more can follow the links.
• llms-full.txt — Ideal when you want agents to have deep context without making multiple requests. Commonly used for developer documentation where an LLM coding assistant needs the full API reference in one shot.

Sites like Supabase take a modular approach: their llms.txt links to topic-specific files like /llms/guides.txt and /llms/js.txt, letting agents load only what they need.

4. How to Create One

Static Sites (HTML, Hugo, Jekyll)

The simplest approach: create a file called llms.txt in your site's root directory (or public/ folder). It will be served at /llms.txt automatically.

# My Company

> We help developers ship faster with open-source tools.

## Docs

- [Getting Started](https://mycompany.com/docs/start.md): Installation and setup
- [API Reference](https://mycompany.com/docs/api.md): REST API endpoints

Next.js

In Next.js App Router, create a route handler that returns plain text:

export async function GET() {
  const content = `# My SaaS App

> A platform for building AI-powered workflows.

## Docs

- [Quickstart](https://myapp.com/docs/quickstart.md): Get up and running in 5 minutes
- [API Reference](https://myapp.com/docs/api.md): Full REST API docs
- [Webhooks](https://myapp.com/docs/webhooks.md): Event-driven integrations

## Optional

- [Changelog](https://myapp.com/docs/changelog.md)
`;

  return new Response(content, {
    headers: { "Content-Type": "text/plain; charset=utf-8" },
  });
}

WordPress

Add a rewrite rule or use a plugin. The simplest manual approach is to drop an llms.txt file into your WordPress root directory. For dynamic generation, add this to your theme's functions.php:

add_action('init', function() {
    add_rewrite_rule('^llms\.txt$', 'index.php?llms_txt=1', 'top');
});

add_filter('query_vars', function($vars) {
    $vars[] = 'llms_txt';
    return $vars;
});

add_action('template_redirect', function() {
    if (get_query_var('llms_txt')) {
        header('Content-Type: text/plain; charset=utf-8');
        echo "# " . get_bloginfo('name') . "\n\n";
        echo "> " . get_bloginfo('description') . "\n\n";
        echo "## Pages\n\n";
        $pages = get_pages(['sort_column' => 'menu_order']);
        foreach ($pages as $page) {
            echo "- [" . $page->post_title . "](" . get_permalink($page) . ")\n";
        }
        exit;
    }
});

Shopify

Shopify doesn't natively support custom root-level files. Your options:

• Use a reverse proxy (Cloudflare Workers, Vercel edge functions) to serve /llms.txt from a separate source
• Create a page at /pages/llms-txt and set up a redirect
• Use a tool like Inlay that hosts and serves the file for you

Documentation Frameworks

Several documentation tools have first-class llms.txt support:

• VitePress — Use vitepress-plugin-llms for automatic generation
• Docusaurus — Use docusaurus-plugin-llms
• Drupal — The LLM Support recipe provides full llms.txt support
• nbdev — Generates .md versions of all pages by default

5. Real-World Examples

Hundreds of sites now serve llms.txt. The llms.txt directory lists over 400 adopters. Let's analyze some notable ones.

Stripe

Stripe's docs.stripe.com/llms.txt is a masterclass in organization. It lists key documentation pages with concise descriptions, linking to .md versions of each page:

# Stripe Documentation

## Docs
- [Testing](https://docs.stripe.com/testing.md): Simulate payments to test your integration.
- [API Reference](https://docs.stripe.com/api.md)
- [Webhooks](https://docs.stripe.com/webhooks.md): Listen for events on your webhook endpoint.
...

What they do well: Every link has a clear, actionable description. They link to Markdown versions (.md) of each page, not HTML.

Vercel

Vercel's llms.txt covers their full documentation hierarchy — from getting started to advanced deployment configuration. It's structured as a deep nested outline:

# Vercel Documentation

[Vercel Documentation](https://vercel.com/docs): Vercel is the AI Cloud...

- [Getting Started](https://vercel.com/docs/getting-started-with-vercel)
  - [Projects and Deployments](https://vercel.com/docs/.../projects-deployments)
  - [Use a Template](https://vercel.com/docs/.../template)
...

What they do well: Nested structure mirrors the actual docs hierarchy, making it easy for LLMs to understand relationships between pages.

Supabase

Supabase takes a modular approach — their root llms.txt is minimal, pointing to topic-specific files:

# Supabase Docs

- [Supabase Guides](https://supabase.com/llms/guides.txt)
- [Supabase Reference (JavaScript)](https://supabase.com/llms/js.txt)
- [Supabase Reference (Dart)](https://supabase.com/llms/dart.txt)
- [Supabase Reference (Python)](https://supabase.com/llms/python.txt)
...

What they do well: SDK-specific files mean an LLM working on a Python project only loads the Python reference. Efficient context use.

Cursor

The AI code editor's llms.txt is highly structured, with sections for getting started, agent features, context management, and customization — all linking to .md versions:

# Cursor Documentation

## Get Started
- https://cursor.com/docs.md
- https://cursor.com/docs/get-started/quickstart.md

## Agent
- https://cursor.com/docs/agent/overview.md
- https://cursor.com/docs/agent/modes.md
...

FastHTML (the original)

The project where llms.txt was born. FastHTML's implementation is the reference example in the spec itself, complete with the blockquote summary and the ## Optional section:

# FastHTML

> FastHTML is a python library which brings together Starlette, Uvicorn,
> HTMX, and fastcore's `FT` "FastTags" into a library for creating
> server-rendered hypermedia applications.

Important notes:
- Although parts of its API are inspired by FastAPI, it is *not* compatible
  with FastAPI syntax
- FastHTML is compatible with JS-native web components and any vanilla JS
  library, but not with React, Vue, or Svelte.

## Docs
- [FastHTML quick start](https://fastht.ml/docs/.../quickstart_for_web_devs.html.md)

## Optional
- [Starlette full documentation](https://gist.githubusercontent.com/...)

6. Why It Matters for AI Agents

If you're not thinking about AI agents yet, you should be. Here's why llms.txt is becoming as essential as robots.txt was in the 2000s.

AI agents are the new search crawlers

ChatGPT, Claude, Perplexity, Gemini — these aren't just chatbots anymore. They're agents that browse the web, read documentation, and synthesize information for users. When someone asks “How do I set up Stripe webhooks?”, the AI doesn't just recall training data — it actively fetches and reads Stripe's docs.

A well-structured llms.txt file is like a VIP entrance. Instead of the agent struggling to parse your HTML, it gets a clean, curated overview immediately.

Context windows are precious

Even with models supporting 100K+ token windows, context is a limited resource. An agent juggling multiple documentation sources can't afford to waste 50,000 tokens on your site's navigation HTML. llms.txt gives agents the minimum effective context — the most information per token.

Coding assistants and IDEs

This is where llms.txt adoption has been fastest. Tools like Cursor, Windsurf, and GitHub Copilot use documentation context to provide better code suggestions. When a developer adds a library, the IDE can fetch its llms.txt to instantly understand the API surface, key concepts, and gotchas.

RAG and search-augmented generation

Retrieval-augmented generation (RAG) systems — the backbone of tools like Perplexity — are more effective when source documents are clean Markdown. llms.txt provides a pre-curated index that helps RAG pipelines select the most relevant pages to retrieve.

MCP servers and tool use

The Model Context Protocol (MCP) lets AI agents connect to external tools and data sources. llms.txt complements MCP by providing the readable context that MCP serves as executable tools. Together, they make your site both understandable and actionable by AI. Explore Inlay's MCP Directory →

7. Validation and Testing

Created your llms.txt? Here's how to verify it works.

Manual checklist

1. Visit https://yoursite.com/llms.txt in a browser — it should return plain text
2. Verify it starts with an H1 (# Your Site Name)
3. Check that all linked URLs are accessible and return valid content
4. Confirm the Content-Type header is text/plain or text/markdown
5. Ensure no HTML is mixed into the Markdown

Test with an LLM

The best validation is to actually use it. Paste your llms.txt content into ChatGPT or Claude and ask questions about your site. If the model can accurately describe your product, find the right documentation pages, and understand your API — it works.

Programmatic validation

Use the official llms_txt2ctx CLI to parse and expand your file. If it parses without errors, you're spec-compliant:

pip install llms-txt
llms_txt2ctx llms.txt          # Parse and expand to context file
llms_txt2ctx llms.txt --optional  # Include the Optional section

Use Inlay's audit

Inlay's free AI readiness audit checks your llms.txt alongside your overall AI discoverability — including MCP servers, structured data, and more. It flags issues like broken links, missing descriptions, and format errors automatically.

8. Automate It with Inlay

Creating and maintaining a llms.txt file manually works for small sites. But as your site grows — new pages, updated docs, deprecated features — keeping it in sync becomes another maintenance burden.

That's where Inlay comes in. Inlay crawls your site, understands your content structure, and automatically generates and hosts your llms.txt. It stays up to date as your site changes, and serves it alongside an MCP server that gives AI agents structured tool access to your content.

The result: your site becomes a first-class citizen in every AI conversation — discoverable, understandable, and actionable. No manual file management required.

Run a free AI readiness audit →