Six Places You Can Watch for AI Agents, and What Each One Can't See
We've already written about why your analytics can't see AI agents. Short version: agents don't execute JavaScript, so the tag never fires, and every client-side tool is blind by construction.
That's the easy half. The harder question is the one nobody writes about: if not the tag, then where? Because the answer isn't one place, and the position you pick determines which questions you're able to answer at all.
Two things get conflated constantly here: where you stand, and how sure you are. They are independent, and you need to solve both.
Vantage: where the agent is observable
Every one of these sits somewhere in the path a request takes through your infrastructure. The further out you stand, the more you see, and the less of it runs your code.
| Vantage point | Sees | Blind to | Status codes |
|---|---|---|---|
| CDN / edge worker | Every request, including ones answered before your origin | Traffic on hosts you don't front | Yes |
| Platform log drain | Everything the platform routed | Real time; anything hosted elsewhere | Yes |
| App middleware | Requests entering the application | Edge-terminated and cached responses | No |
| Framework / server | Application-level requests | Anything served before it reaches you | Yes |
| MCP server | Protocol-level tool calls | Web traffic entirely | n/a |
| Log file import | Whatever is in the file | Live traffic | Yes |
The middleware problem
One row in that table deserves its own section, because it trips up almost everyone who builds this themselves.
Application middleware runs before the response exists. It can read the incoming request, classify the user-agent, and log that an agent asked for /docs/webhooks. What it cannot tell you is whether that request returned a 200 or a 404, because at the moment middleware runs, nothing has decided yet.
That sounds like a detail. It removes an entire category of insight. The most actionable thing in agent analytics is the gap report: what did agents ask you for that you didn't have? An agent requesting /llms.txt or /.well-known/mcp.json and getting a 404 is a specific, fixable, revenue relevant miss. You cannot see it from middleware. You need a vantage point that observes the completed response.
This is geometry, not tooling. No amount of engineering makes middleware see a response that hasn't happened.
Evidence: how sure you are it was an agent
Separate axis. Once you're standing somewhere useful, the question becomes what you're willing to claim, and the honest answer varies a lot by signal.
| Signal | Confidence | Coverage |
|---|---|---|
| Protocol (MCP call, agent auth, API key) | Certain. It announced itself. | Narrow |
| Session (signed in, linked visitor token) | Strong | Narrow |
Referrer or explicit ?via= | Medium | Medium |
| User-agent string | Weak. Spoofable, declared bots only. | Broad |
| Behavioral (no subresource fetches) | Medium | Broad |
The bottom two rows are where the interesting problem lives. String matching a user-agent works fine for GPTBot, ClaudeBot, PerplexityBot and the rest of the named crawlers, because they identify themselves honestly and have no reason not to.
It does nothing for a coding agent running fetch from a developer's laptop while it decides which SDK to install. That request carries a generic user-agent or none at all. It is one of the highest intent interactions you will ever receive, and by user-agent alone it is indistinguishable from noise.
The signal that reaches it is behavioral. A browser rendering a docs page pulls dozens of subresources: stylesheets, fonts, images, scripts. An agent pulls the HTML and stops. That asymmetry doesn't depend on anyone declaring anything, and it holds regardless of what the user-agent claims.
What this means for how you instrument
No single vantage point covers everything, and that isn't a gap anyone is going to close. An edge worker never sees an MCP call. An MCP server never sees a docs crawl. A log drain never sees the domain you host somewhere else.
So the practical sequence:
- Stand at the edge first if you can. It sees the most, including requests that never reach your origin, and it can report status codes, which unlocks the gap report.
- Add the protocol surfaces next. MCP calls and agent auth are the only signals where you know rather than infer, and they're the ones tied most directly to an agent doing something real.
- Treat user-agent as a floor, not a method. It's cheap and broad and it will systematically undercount exactly the traffic that matters most.
- Baseline before you change anything. Otherwise every result you get afterward is a story you told yourself.
Pickrate installs at each of these positions, which is the reason the gap report works on live traffic rather than only on uploaded logs. But the map above is true whether you use us or build it yourself, and it's worth knowing before you pick a spot to stand.
FAQ
Where can I actually see AI agent traffic?
Anywhere on the server side that touches the request path: your CDN or edge worker, a platform log drain like Vercel's, application middleware, your web framework, your server logs, or an MCP server. Not client-side analytics, because agents don't execute JavaScript, so the tag never fires. The CDN is the single best vantage point because it sees requests that get answered at the edge and never reach your origin at all.
Why can't Next.js middleware tell me what status code an agent got?
Because middleware runs before the response exists. It sees the incoming request and can classify the user-agent, but the status code hasn't been determined yet, so it can't report whether the agent got a 200 or a 404. Only a vantage point that sees the completed response can do that: a CDN worker, a log drain, or a server-side route handler. This is a structural property of where the code runs, not a limitation anyone can patch.
Is a user-agent string enough to detect AI agents?
It catches the declared ones and misses a growing share. GPTBot, ClaudeBot, PerplexityBot and the other named crawlers identify themselves honestly, so string matching works. But a coding agent running fetch or curl from a developer's laptop sends a generic user-agent or none at all, and looks like ordinary traffic. Catching those requires behavioral signals, most usefully the subresource gap: a browser pulls dozens of assets per page, an agent pulls the HTML and stops.
Do I need to install something on every surface to measure agent traffic?
No, but no single vantage point covers everything, so coverage is a real tradeoff. An edge worker sees your web traffic including edge-terminated requests but never sees MCP calls. An MCP server sees protocol-level interactions but no web traffic. A log drain sees whatever your platform routed and nothing hosted elsewhere. Pick the position that covers the surface you care most about, and add others when the gaps start costing you answers.
See your Pick Rate
Check how often AI agents pick your tool — free, no account needed.
Check your tool →Keep reading
Serve Both: /index.md and Accept: text/markdown
Content negotiation is the correct way to serve markdown to agents. It is also the one most agents don't use. Cloudflare tested seven; three sent the header. Here's why you need the ugly URL fallback too, and why that stays true no matter how good the new discovery standards get.
We Scored 124 Dev Tools on Agent Readiness. It Barely Predicts Who Agents Pick.
We ran Cloudflare's Agent Readiness score against every tool on the Pickrate leaderboards, then joined it to measured Pick Rate. The correlation is 0.19. Stripe scores 4 out of 16 and wins 80% of payments trials. Telnyx scores 14 out of 16 and wins nothing.