llms.txt vs robots.txt vs schema.org: Which Actually Controls How AI Reads Your Site

Three files, three different jobs, and one of them does nothing you think it does.

In any conversation about making a site AI-ready, these three names arrive together as though they were three options for one task. They are not. One controls access, one clarifies meaning, and one is a suggestion.

Each in one sentence

• robots.txt declares which paths should not be crawled.
• schema.org tells a machine what type of content this page is and what it relates to.
• llms.txt points language models at the site's key sources.

Side by side

Row three is the one that matters. None of these files is an enforcement mechanism. robots.txt is a polite request that well-behaved crawlers honour and others ignore.

The common mistake: treating llms.txt as access control

I have seen this repeatedly: a team writes an llms.txt believing they are declaring what models are permitted to use.

llms.txt forbids nothing. It is a map, not a lock.

If you want part of a site left uncrawled, robots.txt is the tool — and even that only works for crawlers that have chosen to comply. If you want content genuinely unavailable, the only real answer is authentication.

The distinction is practical, not semantic. A team that believes llms.txt protects its content makes bad decisions about what to publish.

What to put in each

robots.txt. Keep it short. Block operational paths with no public value: login, checkout, infinite filter combinations, internal search results. Declare the sitemap. If you want a position on AI crawlers, this is where it goes.

schema.org. Mark the primary type of each page and stop there. For an article, the author and the updated date. For a service page, the provider and the service type. Over-marking causes more harm than under-marking.

llms.txt. A short list of your most important sources, each with one line describing which question that page answers. The descriptions are the point; without them the list is just another sitemap.

The fourth option nobody names

A structured JSON profile of identity and expertise — something like ai-profile.json — does what none of the three does: it makes explicit, checkable claims about who you are, what you know, and where that knowledge comes from.

It is not standardised and guarantees nothing. But it costs an hour, and unlike the other three it produces new semantic content rather than labelling existing content.

A minimal setup covering all three

1. In robots.txt, block only operational and valueless paths, and declare the sitemap. Done once.
2. Add Article schema with author and date to every post. Usually one template change, not per-page work.
3. Write an llms.txt with five to fifteen key sources, each with a one-line description.
4. Add a structured identity profile and link to it from the homepage.

Total: under a day. All of it is infrastructure — the hard part is still the content itself.

Why robots.txt is no longer sufficient

robots.txt was designed for a world where a crawler took your content in order to link to it in search results. The bargain was clear: you granted access, the engine returned traffic.

Answer engines change that bargain. Content is read, an answer is composed, and the user may never reach your site. robots.txt has no vocabulary for the distinction — you cannot say "yes for indexing, no for training, yes for answering with attribution".

Several crawlers now identify separately by purpose, which makes some of this expressible. That too is a voluntary convention, and coverage is partial. The decision genuinely under your control is neither technical nor in these files: what you publish openly, and what you keep behind authentication.

The order of work, starting from zero

If I were starting today, the order would be:

1. Three to five structured articles with tables, definitions and numbers. This is the only part that genuinely takes time.
2. Basic schema on those — author and updated date.
3. An llms.txt pointing at those same three to five pieces.
4. A structured identity profile.
5. A review of robots.txt, only to block valueless paths.

Most teams run this list backwards, because steps four and five fit in an afternoon and step one does not.

A worked example: the wrong order

A product team spent four days implementing all three carefully, with complete markup and a detailed llms.txt listing forty entries.

Three months later there were almost no citations. The cause was not subtle: the forty pages listed in llms.txt were all sales pages. Not one carried a definition, a table, or an attributable number.

The files were correct. What they pointed at was not quotable.

The team eventually reversed the order: first six structured articles with tables and definitions, then an llms.txt update pointing at those. The infrastructure work took half a day, and this time it did something.

How to tell it is working

All three of these files fail silently. A syntax error in robots.txt or a markup mistake raises nothing.

Three checks worth running, each a few minutes: validate the markup once per page type with a structured-data validator; test robots.txt against a few real paths to confirm it blocks what you think it blocks; and open llms.txt to verify every link in it is still live.

The third is the most likely to be broken, because site structure changes over time and that file is rarely revisited.

Limits worth stating plainly

• None of these guarantees ranking, and any claim otherwise is a sales pitch.
• llms.txt has no uniform adoption yet and may be superseded within two years.
• Incorrect markup is worse than none, because it reads as carelessness.
• None of them makes weak content citable.

Should I block AI crawlers?

It depends on the business model. If your revenue comes from content traffic, there is a real argument. If content exists for credibility and lead generation, blocking removes you from where prospects are now searching.

Where does llms.txt go?

At the domain root, like robots.txt. Some sites also publish a full-text variant containing the complete content of key sources.

How often should these be updated?

robots.txt rarely. Schema when the template changes. llms.txt whenever a new pillar piece ships — in practice, every few months.

Closing

robots.txt says where not to go. schema.org says what this is. llms.txt says where to start. None of them says cite this — the content decides that.

A live example of all three, alongside a structured profile, sits on this site's machine-readable page.