The Rise of llms.txt: What Common Crawl Discovered About AI-Friendly Websites
The web is gaining a new type of file designed specifically for the age of artificial intelligence. Known as llms.txt, the proposed convention is often compared with robots.txt, but its purpose is quite different: instead of primarily telling crawlers what they can or cannot access, llms.txt is intended to provide AI systems with a cleaner guide to a website's most important information.
A recent analysis by Common Crawl offers one of the largest looks yet at how websites are actually using the format. The organization examined 584,107 llms.txt files collected during its July 2026 crawl and discovered that many websites are generating the files automatically, while some are even attempting to use them as if they were access-control files.
What Exactly Is llms.txt?
llms.txt is a proposed web convention that places a Markdown file at a website's root, such as:
https://example.com/llms.txt
The idea is relatively simple. A website can use the file to identify important pages and provide short descriptions that are easier for AI systems and agents to understand.
A typical file may contain:
# Example Website
> A short description of the website.
## Documentation
- [Getting Started](https://example.com/start)
- [API Reference](https://example.com/api)
## Guides
- [Tutorials](https://example.com/tutorials)
This makes llms.txt more like a machine-readable table of contents than a traditional crawler-control file.
The proposal has gained attention as AI-powered search engines, coding assistants and autonomous agents increasingly need efficient ways to locate useful information on websites.
Common Crawl Examined More Than Half a Million Files
The scale of the latest Common Crawl investigation is significant.
Researchers analyzed 584,107 llms.txt files from the July 2026 crawl. Their findings suggest that adoption is growing, but much of it is being driven by website platforms and plugins rather than individual developers manually creating the files.
One of the most notable findings was that approximately 68% of the files originated from plugins or templates.
Wix alone accounted for a large share of the files examined, while other SEO and website-management tools also automatically generated llms.txt content.
This means the growing number of llms.txt files does not necessarily indicate that website owners have deliberately developed an AI-content strategy.
In many cases, the file may simply have appeared because a platform enabled the feature.
Many Files Don't Contain Links
The proposed format is intended to help AI systems find useful pages through structured links. However, Common Crawl discovered that a substantial number of files contain no links.
Around 22% of the analyzed files had no links at all.
That raises an important question: how useful is an llms.txt file if it doesn't actually point an AI system toward the website's important resources?
The answer depends on how the file is constructed.
A carefully written file can function as a concise map of a website. An automatically generated or nearly empty file may provide little practical value.
Some Websites Are Treating llms.txt Like robots.txt
Perhaps the most interesting discovery involves crawler restrictions.
Some website owners appear to believe that llms.txt can be used to allow or block AI crawlers in the same way that robots.txt is used.
Common Crawl found 1,570 files that mentioned specific crawlers, including 32 that appeared to deny access to CCBot, the crawler operated by Common Crawl.
But there is an important technical distinction.
robots.txt controls crawler access
The traditional robots.txt mechanism is designed to communicate crawling rules.
For example:
User-agent: CCBot
Disallow: /
A crawler that follows the robots exclusion protocol can use this information when deciding whether to fetch content.
llms.txt does not work that way
llms.txt is intended primarily to describe and organize information for AI systems.
It does not automatically grant or deny access.
Common Crawl checked the robots.txt files associated with sites that appeared to block CCBot through llms.txt. Of the 31 sites it could evaluate, none actually blocked CCBot outright through robots.txt.
This is an important lesson for website owners: putting "AI crawler blocked" inside llms.txt does not necessarily stop a crawler.
Why robots.txt Still Matters
The two files serve different purposes.
| File | Main purpose |
|---|---|
robots.txt |
Communicate crawler access rules |
llms.txt |
Provide an AI-friendly content map |
sitemap.xml |
Help crawlers discover URLs |
llms-full.txt |
Provide a larger AI-oriented content representation |
Think of the difference this way:
robots.txt = "What may you crawl?"
llms.txt = "Here is what is important on my website."
sitemap.xml = "Here are the URLs on my website."
Keeping these roles separate can prevent significant confusion.
Templates Are Driving Adoption
Another important finding is the growing role of automated tools.
Instead of website owners manually writing llms.txt, many content-management systems and plugins are producing them automatically.
Common Crawl reported that about two-thirds of the files it analyzed were generated through plugins or templates.
This helps explain why adoption numbers can vary dramatically between studies.
For example, separate research from Ahrefs found that 28% of 137,000 domains in its Web Analytics sample published a valid llms.txt file. However, that sample is more likely to contain technically sophisticated websites than the entire internet.
Other datasets produce considerably lower figures.
The takeaway is that there is currently no single universal adoption number for llms.txt.
Does llms.txt Actually Improve AI Visibility?
This is where website owners should be cautious.
Having an llms.txt file does not automatically mean that an AI system will use it.
Research has found that major AI providers have not universally committed to treating llms.txt as a standard crawler protocol.
That means publishers should not assume that adding the file will immediately result in:
- More AI citations
- Higher search rankings
- More website traffic
- Better visibility in ChatGPT
- Better visibility in Google AI results
The technology is still developing.
However, there may still be a practical reason to create one: it is relatively inexpensive to provide a clean, structured summary of a website's most valuable content.
A Potential Benefit for Documentation Websites
llms.txt may be particularly useful for websites containing technical documentation.
Imagine a software company with hundreds of pages covering:
- API references
- Installation instructions
- Tutorials
- SDK documentation
- Authentication
- Troubleshooting
- Examples
An AI coding assistant could potentially benefit from a concise file pointing toward the most authoritative documentation.
This is one reason developer-focused companies and documentation platforms have shown considerable interest in the format.
For these websites, an llms.txt file can act as a curated entry point rather than another generic SEO document.
There Is Also a Security Concern
Common Crawl discovered a small number of files containing text that appeared to be attempts at influencing an AI model.
Ten files matched its strictest prompt-injection detection test. After manual examination, researchers identified four genuine cases, while several others were false positives or examples used for research and demonstrations.
The numbers are extremely small compared with the hundreds of thousands of files analyzed.
Nevertheless, the discovery highlights an important principle:
AI systems should not automatically trust instructions contained inside an llms.txt file.
A file intended to describe website content could potentially contain malicious or misleading instructions.
AI agents therefore need to distinguish between:
- Information about a website
- Instructions directed at the AI
- Actual crawler permissions
- Potentially malicious content
This distinction becomes increasingly important as AI agents gain the ability to browse and take actions on the web.
What Website Owners Should Do
For most website owners, the safest approach is relatively straightforward.
1. Don't treat llms.txt as a replacement for robots.txt
If you want to communicate crawler restrictions, use the appropriate robots.txt mechanism.
2. Keep llms.txt useful
If you publish one, include your most important pages and documentation rather than filling it with promotional material.
3. Keep it updated
A broken or outdated content map could be worse than having no file at all.
4. Don't expect instant SEO benefits
There is currently insufficient evidence to treat llms.txt as a guaranteed ranking or traffic strategy.
5. Use clear descriptions
Short explanations can help humans and machines understand why a particular page matters.
6. Keep security in mind
AI systems should never blindly obey instructions simply because they appear inside an llms.txt file.
The Bigger Picture
The emergence of llms.txt reflects a much larger change taking place across the internet.
For decades, websites were primarily designed for humans, search engines and traditional crawlers. The rise of generative AI introduces another consumer of web information: AI agents that need to understand content rather than simply index it.
That creates a new challenge.
A traditional search engine can crawl thousands of pages and build an index. An AI agent may instead need to identify the most authoritative page, understand its context and retrieve only the information required to answer a question or complete a task.
A structured content map could eventually become useful in that environment.
But Common Crawl's research also demonstrates why adoption alone isn't enough. If websites generate thousands of low-quality files through templates, misunderstand the purpose of the format or attempt to use llms.txt as an access-control mechanism, the value of the system becomes much less clear.
Final Thoughts
The latest Common Crawl analysis provides an important reality check on the rapidly growing llms.txt conversation.
More than half a million files were examined, and the results show a format that is gaining adoption but remains inconsistent in practice. Many files are automatically generated, a significant percentage contain no links, and some websites mistakenly treat llms.txt as an alternative to robots.txt.
For now, llms.txt should be viewed as an emerging convention rather than a universal internet standard.
Its long-term importance will ultimately depend on whether AI search engines, agents and other machine-reading systems actually use the information it provides.
One thing is already clear, though: as AI becomes a major way people discover information online, websites are beginning to create new machine-readable layers specifically for AI. llms.txt may be an early experiment in what that future web looks like.
