Serve Both: /index.md and Accept: text/markdown

There is a correct way to serve markdown to an AI agent, and there is a way that works. You should do both, and it is worth understanding why that is not a contradiction.

The correct way is content negotiation. An agent sends Accept: text/markdown with its request, your server sees it, and returns the markdown representation of that same URL. One canonical address, two representations, nothing duplicated. This is what HTTP was designed to do, and it has worked this way since 1997.

The way that works is /index.md. Ugly, redundant, a second URL for a document you already have. And necessary, because most agents never send the header.

The number that settles it

In February 2026, Cloudflare tested seven AI agents to see which ones request content with Accept: text/markdown by default. Three did: Claude Code, OpenCode, and Cursor. The other four asked for a URL like any other client.

Sit with that ratio for a second. These are the agents whose entire job is reading documentation. If fewer than half of the most technical agents in the market send the header, the population of general assistants, retrieval fetchers, and scrapers reading your site sends it far less often than that.

So if content negotiation is your only markdown path, you have built a feature for a minority of your most sophisticated readers and nobody else. The elegance is real. The coverage is not.

What each one actually solves

These are not two implementations of the same idea. They fail in different directions, and that is precisely why you want both.

Content negotiation solves identity. There is one URL for the document. Your citation, your sitemap, your canonical tag, and the link someone pastes into Slack all point at the same address. An agent that knows to ask gets markdown from the same place a human gets HTML. Nothing forks.

The .md URL solves reachability. An agent that has never read your llms.txt, does not know your conventions, and does not send an Accept header can still guess. It saw example.com/docs in a search result, it appends four characters, and it gets clean markdown. No negotiation, no discovery document, no prior knowledge.

Identity without reachability is a document only insiders can open. Reachability without identity is a duplicate content problem. Together they are one document with two front doors, which is what you want.

The root is the one that gets missed

Most sites that add markdown twins map them per page and forget the site root. What is the markdown twin of /? Appending .md to a bare slash gives you /.md, which is not a URL anybody wants to write. So the convention landed on /index.md, borrowed from the oldest habit on the web.

That is worth getting right, because the homepage is the URL an agent is most likely to have and least likely to find useful as HTML. It is the most markup-heavy, most navigation-heavy, least content-dense page you serve. It is also the first thing an agent fetches when someone asks what your company does.

We shipped /index.md on this site after noticing the same gap in our own implementation. Content negotiation had worked at the root since launch. The URL fallback did not exist, so any agent that did not send the header got 191KB of HTML instead of a few hundred lines of markdown that answers the question directly.

The three headers that keep it clean

The objection to serving the same content at two addresses is duplicate content. Three response headers make it a non-issue.

Vary: Accept on the negotiated response tells caches that this URL has more than one representation and which request header selects between them. Without it, a CDN will happily serve cached markdown to a browser, or cached HTML to an agent that asked for markdown. This is the header people forget, and the resulting bug is maddening to reproduce because it depends on who warmed the cache.

X-Robots-Tag: noindex on the markdown rendering keeps search engines indexing the HTML page as the canonical document. The markdown is a machine representation, not a second page competing with the first.

Content-Type: text/markdown; charset=utf-8 because a client that asked for markdown should be told it got markdown. Serving it as text/plain works by accident and breaks anything that branches on the type.

Why the new standards don't retire the old paths

Here is the part worth arguing about. The discovery layer is getting genuinely good. Agentic Resource Discovery gives you a signed catalog of your agentic resources. MCP server cards let a client inspect a server before opening a transport. RFC 9727 API catalogs enumerate every API you publish. RFC 8288 Link headers announce all of it from a single HEAD request.

It is tempting to read that stack and conclude the crude stuff is transitional. Once every agent reads your catalog, who needs a guessable URL?

That reasoning has a hole in it: discovery and delivery are different problems. Every one of those standards answers where things are. None of them changes what happens when a client issues a plain GET for a URL it already has. And clients constantly have URLs they did not discover from you: pasted into a prompt, quoted in a citation, embedded in training data, copied from a forum post in 2024, returned by a search API that has never heard of your catalog.

There is a second reason, less philosophical and more decisive. Standards adoption is not gated on the standard being good. It is gated on every client implementing it, and clients update on their own schedule or never. Cloudflare's own numbers make the point: robots.txt, a convention from 1994 with no formal standing for most of its life, is on 78% of sites. Content Signals, which is genuinely better designed, is on 4%. The gap is not quality. It is thirty years.

The agents reading you today were written against the web as it existed when they were built. Some of them will not be updated again. Serving the old path costs you one route handler. Not serving it costs you every reader who arrived by any means other than the one you designed for.

What to actually do

Serve markdown by content negotiation on every content URL. Serve the same markdown at path.md for every one of those URLs. Serve the site root at /index.md. Set Vary: Accept, X-Robots-Tag: noindex, and a real Content-Type. Then list the whole thing in your llms.txt so an agent that does read discovery documents does not have to guess at all.

That is roughly an afternoon, and it is the highest-leverage afternoon in agent readiness, because it is the only work that pays off for both the agents that read your standards and the ones that never will.

Whether any of it changes who agents actually pick is a separate question, and a harder one. That is what Pick Rate measures.