llms.txt and AI crawlers: opening your site to models
Language models read websites with their own crawlers. Many companies block them out of habit, along with all suspicious traffic — and then wonder why they are never mentioned in answers. Let us look at how to manage this deliberately.
llms.txt is a short file where you describe in plain language what the company is and where things live. It is not mandatory and guarantees nothing, but it costs ten minutes and helps models avoid mistakes. More important: do not block AI crawlers in robots.txt if you want to appear in their answers.
In this article8 sections
Who visits your site
Besides the familiar Google and Yandex robots, AI services send crawlers of their own. They fall into two kinds, and the distinction matters.
| Kind | Why it visits | Should you allow it |
|---|---|---|
| Training crawler | Collects text to train models | Your call |
| Real-time search crawler | Reads the page to answer a person's question now | Usually yes |
The second kind determines whether you get named in an answer. Block it and you drop out of the channel for certain.
A common situation: an administrator blocked everything unfamiliar in robots.txt, AI crawlers included. Formally safe, in practice a voluntary refusal of a growing source of customers.
How to allow access
Control goes through the same robots.txt file where you set rules for ordinary search engines. AI crawlers get their own rules by name.
The ones usually worth allowing for a business: the OpenAI, Anthropic and Perplexity crawlers, plus the extended Google and Apple crawlers. Names change, so the list is worth reviewing every six months.
- Permission is written explicitly — by crawler name with an Allow directive.
- Access to technical folders is not needed, only to content pages.
- Pages with personal data and service sections are closed, just as they are for ordinary robots.
What llms.txt is
This is a young standard: a text file in the root of the site where a company describes itself for language models in a free but structured form. Something like a short brief: what we do, which services, what prices, where we are, where to go for details.
Formally, no service guarantees support for it yet. In practice it costs almost nothing — it is one text file — and it is useful as a clear, non-contradictory source of facts about the company.
What to write inside
- A short description of the company: what it does, for whom, where it works.
- Key facts to quote: service lines, prices, timing, cities.
- A list of the main pages with a note on what is on each.
- Honest limitations: what you do not do and what you do not guarantee.
- Contacts.
The main rule is the same facts as on the site. The file must not contradict the pages, otherwise it hurts instead of helping.
How to tell whether you blocked yourself by accident
This happens more often than you would think: the rule was added long ago, the reason was forgotten, and the channel is quietly not working.
- Open your robots.txt — it sits at yoursite.com/robots.txt.
- Look for Disallow lines under AI crawler names or under the general User-agent.
- Check that access is not blocked at server level — by agent name or by country.
- Ask your hosting provider whether its bot protection blocks these requests.
That last point is the most common hidden cause. Systems that protect against unwanted traffic often block AI crawlers by default, and robots.txt shows no sign of it.
What to put in llms.txt: a structure
| Section | What it contains |
|---|---|
| Heading | Company name and one line on what it does |
| Key facts | Service lines, prices, timing, cities, legal form |
| Site sections | A list of main pages with a note on what is on each |
| Limitations | What you do not do and what you do not guarantee |
| Contacts | Email, messenger, office addresses |
One or two pages of text. The file should read like something written for a person: if a person understands it, a model will too.
What this file does not do
- It does not replace the content of the site: the model still looks at the pages.
- It does not guarantee a mention — it is not an application to be included in answers.
- It does not affect ordinary search.
- It does not protect your text from being used: that needs technical restrictions, not a declaration.
What else affects how accessible your site is to models
- The content must be in the HTML. If text only appears after scripts run, some crawlers will not see it.
- The page must load quickly. Crawlers have a limited waiting time.
- There must be no blocking by region or agent, if you want to be read.
- A sitemap helps crawlers find pages instead of only following links.
When access should be blocked
There are legitimate reasons not to let training crawlers in, and it is a normal decision.
- Unique paid content that you sell.
- Original material you do not want fed into training.
- Users' personal data — which should not be publicly accessible at all.
Real-time search crawlers are usually still left open: they bring people rather than take text.
Frequent questions
Is llms.txt mandatory?
No, it is a voluntary and young standard with no guaranteed support. But it costs one file and delivers enough to be worth having.
Will my content be stolen?
Text on an open website is available to everyone anyway. If the material is valuable and sold, it must be closed behind a login rather than protected by a request to robots.
How do I know whether AI crawlers visit me?
From the server logs: agent names are visible there. If you have no access to the logs, ask your hosting provider — such statistics usually exist.
Does this affect ordinary SEO?
Allowing AI crawlers has no effect on Google and Yandex rankings. It is a separate channel with its own rules.
Do AI crawlers slow the site down?
On an ordinary site the load is unnoticeable. If there are too many requests, their rate can be limited on the server side without blocking them completely.
Should llms.txt be referenced in robots.txt?
Not required — the file is looked for at the standard root address. But mentioning it with a link does no harm and breaks no rules.
The materials answer general questions. We will look at your specific case — free and without obligation.