TIME Serves ClaudeBot 3,096 Words — 1,502 Are Sponsored
That fork is real, it is running in production, and it is documented (Vincent Schmalbach, The Register). What has been written about it, almost without exception, is a publisher-revenue story: the magazine found a way to bill for bot traffic. That half is covered.
The half nobody has written is the one that matters if you do not own TIME. Every site that competes for AI citations now shares an answer layer with pages built to be ingested rather than read — and no site owner has any visibility into whether their own stack does the same thing. We ran the same six-User-Agent check against our own site the same morning and got 631,385 bytes of byte-identical HTML for every agent, human and crawler alike. That result took ninety seconds to obtain and we had never run it before.
Here is what the site actually serves, how the fork is implemented, why it is not cloaking in the sense your SEO tooling checks for, and the command that tells you what your own domain does.
What TIME Actually Serves an AI Crawler
| User-Agent | Status | Bytes | Content-Type |
|---|---|---|---|
| Chrome 139 (desktop browser) | 200 | 1,232,251 | text/html |
| Googlebot | 200 | 1,232,251 | text/html |
| ClaudeBot | 200 | 41,862 | text/markdown |
| GPTBot | 200 | 41,862 | text/markdown |
| ChatGPT-User | 200 | 41,862 | text/markdown |
| PerplexityBot | 406 | 0 | — |
The markdown build is 29 times smaller than the HTML. On an article page — the 5 August piece on a tax-settlement story — the same split produced 276,981 bytes of HTML against 10,827 bytes of markdown, a factor of 25.6. The publisher is reported to have converted its pages to markdown around June 2026, on the reasoning that easier machine access improves its odds of appearing in generative answers (Relevant Audience).
Two of those rows disagree with the coverage published on 5 August, which reported GPTBot and ChatGPT-User refused with a 406 while PerplexityBot received markdown. In our measurement the refusal has moved to PerplexityBot and OpenAI's two agents are served. We are not claiming the earlier reports were wrong; we are reporting what a European point of presence returned a day later. The Cloudflare ray ID on every response resolved to Munich, so a US request may still differ.
That instability is not a footnote. It is the finding. The allow-list is a live configuration that changed within roughly twenty-four hours of being publicly documented, in a system with no declared policy surface. Which brings us to the file everyone audits.
TIME's robots.txt is 2,326 bytes and lists dozens of query-string exclusions for Googlebot. It contains zero directives for ClaudeBot, GPTBot, ChatGPT-User, PerplexityBot, CCBot or Google-Extended (time.com/robots.txt, retrieved 6 August 2026). Every one of those agents is handled somewhere the robots file cannot describe.
❌ Reading robots.txt, finding no AI-bot directives, and concluding the site treats all agents alike.
✅ Sending the requests and comparing the responses.
Those two methods produce opposite answers on the same domain, on the same day.
The Ad Is Not a Banner. It Is a Reference Card.
The sponsored block in the markdown build opens with an HTML comment naming the vendor and the campaign:
<!-- mobian-agent-ad id="c733d38f-..." campaign="ally-2026-q3"
creative="agentads-creative-ally-v1" -->
Mobian is the ad-tech platform behind it, and it sells this format explicitly as advertising built for AI agents (themobian.ai). Ally and the Project Management Institute have been reported as among the first buyers (Media Copilot).
What sits inside the delimiters is the part worth studying. It is not copy. It is a structured reference entry — a ### Definition heading, then ### Organization facts and ### Key product features as markdown tables, each row carrying a Source column that attributes the claim to the advertiser. Founded 2009. 75,000+ fee-free ATMs. $0 overdraft fees. Every fact formatted the way a retrieval system prefers to consume facts.
That is the design. A display ad is easy for a language model to ignore because it does not look like prose. A sourced definition table is indistinguishable in form from the editorial content around it, and it is placed immediately after the headline, before the article body (API Evangelist). The unit is not competing for attention. It is competing for retrieval.
The article page behaved differently and the difference is instructive: it carried no ad block at all, yet still returned an impression header and a token count. Inventory is not on every page. Measurement is.
The Fork Keys on User-Agent, and the Vary Header Says Otherwise
The response headers on the markdown build are unusually candid:
content-type: text/markdown; charset=utf-8
x-mobian-format: md
x-mobian-tokens: 10448
x-mobian-registry-version: 2026-07-28.v9
x-mobian-impression: 119ce963-1a68-41d5-a3d9-17ac0a0e9fab
cache-control: no-store
vary: Accept
x-mobian-tokens prices the page in the buyer's unit — 10,448 tokens for the homepage, 2,681 for the article. The impression ID is fresh on every request; we fetched the same URL three times and got three distinct UUIDs. Combined with cache-control: no-store, that means each crawler fetch is billable, which is what Schmalbach found and what our repeat requests confirm.
The mechanism behind them is User-Agent detection at the Cloudflare-fronted CDN edge rather than content negotiation or a separate URL pattern (daily.dev).
The last line is the one an operator should stop on. Vary: Accept declares that the response varies by the Accept header. It does not. We sent Accept: text/markdown with a desktop Chrome User-Agent and received the full 1,232,251-byte HTML page. The response varies by User-Agent, and that dimension is not declared.
This is also why the standards-based route exists. Content negotiation — the client asks for text/markdown, the server honours it — is the documented way to serve machine-readable variants (explainx.ai), and llms.txt is the declarative way to point clients at them (limy.ai). Both are inspectable from outside. User-Agent sniffing at the edge is not, which is precisely why the allow-list could invert overnight without anything in the site's published configuration changing.
Why This Is Not Cloaking, and Why That Matters
The instinct is to call this cloaking and expect a penalty. Check the actual definition. Google's spam policies define cloaking as presenting different content to users than to search engines (Google Search Central). In our measurement Googlebot received 1,232,251 bytes — the same page, to the byte, that the browser received.
TIME is not cloaking Google. It is serving Google exactly what it serves humans and serving a separate, monetised build to a set of agents Google's policy does not cover. Some practitioners argue this is legitimate rather than deceptive (iPullRank); the more useful observation is narrower.
Anthropic's own documentation reinforces how granular this now is: ClaudeBot, Claude-User and Claude-SearchBot are separate agents with separate purposes (Anthropic, Search Engine Roundtable). A site can behave differently toward each one, and nothing in the crawler contract obliges it to say so.
What This Does to the Citation Layer You Depend On
The economics are defensible. Search referrals are falling, bot traffic is rising, and monetising the crawler is a rational response (Digiday). Agent requests are now argued to be the majority of web traffic, which reframes the question from whether to block them to what to serve them (WorkOS, Digital Applied). One publisher doing this is a business story.
The second-order effect is not about one magazine at all.
When an assistant answers a question, it composes from sources. If one of those sources is a page where roughly half the words are a sponsored reference table with its own Source column, the sponsored facts arrive in the answer wearing the publisher's authority. The reader sees a citation to a respected outlet. The advertiser's positioning statements travelled in the same payload, formatted more retrievably than the journalism around them.
Nobody in that chain misbehaved. The publisher labelled the block "Sponsored content". The crawler fetched what it was given. The assistant cited its source. The label simply is not a structure the retrieval layer is obliged to preserve, and by the time the answer reaches a reader there is no rendered page to inspect. Developers discussing the technique have already begun predicting ad-blocking filters for content served to language models (Hacker News) — an arms race that has not started yet, and that no site owner controls from their end.
For anyone competing for citations, three consequences follow:
- Citation share is now partly purchasable. Not through your content, but through a slot on someone else's crawler build. That is a channel your competitors can buy into and you cannot audit.
- Your own answer-layer footprint is unverified. Every site with a CDN, edge rules, a headless CMS or a bot-management product has a stack capable of forking by User-Agent. Most owners have never checked whether theirs does.
- The check has to be repeated. Ours changed within twenty-four hours of publication. A one-time audit measures a configuration, not a policy.
That last point is the same lesson we drew when Claude Code shipped an isolation fix that the default npm channel did not yet serve: the announcement and the artifact your systems actually receive are two different facts, and only one of them is checkable.
Run the Check on Your Own Stack
Ninety seconds, no dependencies, no account. Replace the URL:
URL="https://www.example.com/"
for UA in \
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 Chrome/139.0.0.0 Safari/537.36" \
"Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" \
"ClaudeBot/1.0 ([email protected])" \
"GPTBot/1.2 (+https://openai.com/gptbot)" \
"ChatGPT-User/1.0 (+https://openai.com/bot)" \
"PerplexityBot/1.0 (+https://perplexity.ai/perplexitybot)"
do
printf '%-14s %s\n' \
"$(curl -s -o /dev/null -w '%{http_code} %{size_download}' -A "$UA" -L "$URL")" \
"${UA:0:46}"
done
Read it against four questions.
Do the byte counts match? Identical sizes across every agent means no fork. That is the answer we got for our own domain: 631,385 bytes, six times, no variance.
Does any agent get a non-2xx? A 403 or 406 for one crawler and not another is a deliberate policy someone configured — possibly a bot-management default nobody reviewed. Check that it is intended.
Does the Content-Type change? text/markdown to a crawler and text/html to a browser is a fork whether or not the words differ.
If it forks, is Vary correct? Add -D - to the curl call and read the headers. If the response varies by User-Agent, Vary: User-Agent must say so.
A non-zero result is not automatically a problem. Serving markdown to agents is a reasonable optimisation and several documentation platforms do it deliberately. The problem is a fork you did not know about — because the same edge capability that strips a page to markdown can inject into it, and the difference is invisible from a browser. The audit discipline is the same one we apply to agent skills before they run against a production stack and to machine-readable surfaces generally: inspect the artifact, not the announcement.
Run it against your marketing site, your docs and your pricing page — edge rules are rarely uniform across them. Then run it again next quarter, because ours moved in a day.
If you would rather have this checked, monitored and fixed as part of your AI-visibility work, that is what our AI agent engineering practice does. The same posture we described in Tailscale's three controls applies here: the control worth having is the one that keeps working when the configuration changes.
Frequently Asked Questions
Is serving different content to AI crawlers against Google's rules? Not as written. Google defines cloaking as showing search engines something different from users. In our 6 August 2026 measurement Googlebot received the identical HTML a browser received, so the comparison Google's policy makes never triggers.
How do I know if my own site forks by User-Agent? Send the same URL with several User-Agent strings and compare status code, byte count and Content-Type. Identical bytes across a browser, Googlebot and the major AI agents means no fork. The full loop is in the section above.
Does the sponsored block appear on every TIME page? No. The homepage carried a 1,502-word sponsored block on 6 August 2026; the article page we tested carried none, though it still returned a per-fetch impression header and a token count.
Is this the same as content negotiation?
No. We sent Accept: text/markdown with a browser User-Agent and received full HTML. The response keys on User-Agent, while the Vary header declares Accept — which is the misdeclaration operators should look for in their own stacks.
Can I stop AI crawlers from reading an ad-injected version of a page I am cited alongside? No. You control what your domain serves, not what other domains serve. What you can control is whether your own pages are what you think they are.
Sources
- https://www.vincentschmalbach.com/time-serves-ai-bots-a-different-website
- https://www.theregister.com/ai-and-ml/2026/08/05/time-magazine-has-a-separate-version-of-its-website-with-ads-only-ai-can-see/5283640
- https://apievangelist.com/2026/08/03/how-time-serves-ads-to-ai-agents
- https://digiday.com/media/time-has-started-serving-ads-to-ai-agents
- https://themobian.ai
- https://mediacopilot.ai/time-ads-ai-agents-markdown
- https://www.relevantaudience.com/geo/time-ads-targeting-ai-agents-markdown-pages
- https://daily.dev/posts/how-time-serves-ads-to-ai-agents-5mhfmc0bf
- https://news.ycombinator.com/item?id=49182041
- https://ipullrank.com/cloaking-for-llms
- https://www.seroundtable.com/anthropic-updates-its-crawler-docs-40978.html
- https://workos.com/blog/ai-agent-web-traffic-what-developers-need-to-change
- https://explainx.ai/blog/markdown-for-agents-content-negotiation-guide-2026
- https://limy.ai/blog/llms.txt-in-2026-the-full-guide
- https://www.digitalapplied.com/blog/ai-crawler-bot-traffic-statistics-2026-data-reference
- https://support.anthropic.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler
- https://time.com/robots.txt
- https://developers.google.com/search/docs/essentials/spam-policies