Cloudflare, the company that serves around 20% of all websites worldwide (including all our own sites), introduced “Markdown for Agents”. The feature automatically converts your HTML pages to Markdown when an AI crawler requests it. The result: up to 80% fewer tokens, meaning faster and cheaper content processing.
At first glance a logical step. But within a day, critical responses were already appearing from the SEO world. And rightly so, because behind this technical improvement lies a fundamental tension.
Two worlds, two interests
What strikes us: two completely different use cases are being conflated.
On one hand you have developers - or really anyone building or using AI agents to pull information from the web. Think of a developer using Claude Code to fetch API documentation, or a tool that automatically collects product information. For them, Markdown is a blessing: fewer tokens means more room in the context window. We use this ourselves daily in our development work and notice the difference immediately.
On the other hand you have website owners, entrepreneurs and marketers who invest in SEO and increasingly in GEO (Generative Engine Optimization) to be visible in AI platforms like ChatGPT, Gemini and Perplexity. For them, the whole point is that AI crawlers see their complete page: the HTML, the structured data, the context. Everything that contributes to how an AI system understands and cites their content.
The problem arises when solutions for the first group are sold as an SEO strategy for the second group.
Google and Bing are clear: do not do this
Google’s John Mueller left no doubt about it on Bluesky. Asked whether websites should create separate Markdown pages for LLM crawlers, he responded that LLMs have been processing ordinary web pages from the very beginning and handle HTML just fine. Shortly after, he was even more direct:
Converting pages to markdown is such a stupid idea. Did you know LLMs can read images? WHY NOT TURN YOUR WHOLE SITE INTO AN IMAGE?
— John Mueller (@johnmu.com) 2026-02-03T15:40:56.697Z
Colleague Gary Illyes agreed: Google does not support llms.txt and has no plans to do so.
Microsoft echoed the same message. Fabrice Canel of Bing warned that separate versions for bots amounts to double crawl load. Bing will fetch both versions anyway to check similarity. His advice: invest in Schema markup and make your pages as good as possible.
The cloaking risk
SEO consultant David McSweeney demonstrated that Cloudflare’s Accept: text/markdown header is passed through to the origin server. Your server can therefore detect exactly when an AI agent visits, and return a completely different HTML response.
In theory you could show one version to humans and a totally different version to AI systems. That is cloaking, and it has been a violation of Google’s guidelines for years.
Technical SEO consultant Jono Alderson put it well: the moment a machine-specific version of your page exists, you have created two candidate versions of reality. From the outside, an AI system now has to choose which version is the real one.
Why llms.txt is not going to help you as a website owner
This brings us to the broader point: llms.txt is not an SEO or GEO strategy for website owners.
The hype is substantial. The idea sounds attractive: a file that tells AI systems where your most important content is, in clean Markdown format. Platforms like Wix even offer automatic generation. But the facts are sobering:
No major AI platform has officially announced support. Mueller compared llms.txt to the keywords meta tag: a standard that once looked promising but is used by nobody because the content is too easy to manipulate.
Google does not crawl llms.txt for search. The Search team confirmed that it is not used for discovery or rankings.
It creates duplicate content. A Markdown version per page means two versions of the same content. Search engines then have to choose which one to trust, and that uncertainty does not work in your favour.
Mueller’s advice is clear: if AI companies knew that a specific format would produce better results, they would be very loud about it. Those companies are not known for their modesty.
Where llms.txt does make sense
To be fair: for technical documentation and API references it is genuinely valuable. Companies like Anthropic, Stripe and Cloudflare use llms.txt for their developer docs, so that AI assistants can answer code-related questions more quickly. That is exactly the developer use case we mentioned earlier.
Building a SaaS product with a public API? Then llms.txt can be a smart investment. But if you are a law firm, an online shop, or a marketing agency? Your time is better spent on good HTML, structured data, and quality content.
What we recommend
As a website owner: focus on what works. Strong, well-structured HTML pages, relevant Schema markup, fast load times, and content that genuinely adds value. That is what both search engines and AI platforms use to understand and cite you. An llms.txt file is not going to change that.
For development and AI tooling: Markdown optimisations like Cloudflare’s feature are extremely useful. We use it ourselves with AI agents and notice the efficiency gains immediately.
Understanding the difference between these two worlds is crucial. Do not get swept up in the hype around llms.txt as a GEO strategy. The major platforms have spoken: just make your website good. The AI will do the rest.