The Search Engine Has Completely Changed: Cloudflare AI Search, llms.txt, and the New AI-Era SEO Playbook

Published August 6, 2026 · Category: Search Engine Optimization · Reading time: ~14 min

The search engine you grew up with — ten blue links, a pagerank score, a keyword-stuffed title tag — is dead. What replaced it is stranger and more interesting: answer engines that read your site, embed it, rerank it, and quote it inside generated answers. Agents that call /mcp endpoints instead of opening a browser. Crawlers that would rather read one clean Markdown file (llms.txt) than parse your JavaScript-heavy homepage.

This article is a complete, working reference for that new era. It covers:

1. What actually changed in search (SEO → GEO → AI-SEO)
2. Cloudflare AI Search — the announcement, the architecture, the pricing preview, real screenshots
3. llms.txt — the spec, a copy-paste template, and how it relates to robots.txt and sitemaps
4. A practical AI-SEO checklist you can execute this week to get cited by AI answers and rank internationally

Everything below is sourced from primary documentation and screenshots (linked at the bottom). No secondhand paraphrase.

Part 1: What Actually Changed in Search

From indexing pages to understanding sites

Classic Google worked like a librarian: crawl, index, rank by authority signals, return links. The user did the reading.

AI-era search works like a research assistant: crawl, chunk, embed (turn text into vectors), retrieve the best chunks for a query, rerank them, and feed them into a language model that writes the answer. The user never clicks. Your content is consumed as context, not as a destination.

That single shift changes what "ranking" means:

GEO (Generative Engine Optimization) is the term now used for optimizing content to be quoted by generative engines — ChatGPT, Perplexity, Google's AI Overviews, Claude, Copilot. AI-SEO is the broader discipline: making your site legible to both answer engines and the agents that operate on top of them.

Three concrete consequences:

1. Traffic is not the goal anymore — citation is. Being the source inside an AI Overviews box or a Perplexity answer is the new #1 position.
2. Structure beats decoration. LLMs prefer dense, factual, well-headed content. Your FAQ block and comparison tables are now prime real estate.
3. New files matter. robots.txt controlled access. sitemap.xml helped discovery. Now llms.txt guides interpretation — and /mcp endpoints expose your data directly to agents.

The four problems AI search platforms had to solve

Cloudflare's latest AI Search announcement is a useful lens because it names the exact problems every site owner faces in this era:

1. Cost predictability — embedding and reranking every page feels like a token-counting nightmare. Cloudflare's answer: free embedding and reranking with default models, part of their pricing preview.
2. One endpoint across many data sources — real knowledge lives in docs + blog + help center + internal wiki. Cloudflare now gives you a single /search and /mcp endpoint per namespace that fans out across instances.
3. Branding and access control — you can serve search from your own domain (search.example.com/mcp) and lock it with Cloudflare Access.
4. Indexing without a sitemap — most real sites don't have clean sitemaps. The new "Discover" parse mode crawls by following links, and subdomains can be added as separate sources.

Let's look at each one in detail.

Part 2: Cloudflare AI Search — A Search Engine for Your Agents

The pitch

Previously, building semantic search on Cloudflare meant stitching together primitives: Workers AI for embeddings, Vectorize for vectors, R2 for storage, Browser Run for crawling, AI Gateway for model routing. AI Search collapses all of that into one product: point it at your files or websites, and it handles crawling, ingestion, embedding, and retrieval automatically.

The stated goal: "give your agents their own search engine, where they can easily find data to provide better answers for themselves and their humans."

What's new in this release

• Index any collection of data — structured or unstructured, from individual files to websites you own (zone on your Cloudflare account today, more ownership verification methods coming).
• Skip the sitemap — the new Discover parsing option crawls your site by following links (powered by Browser Run's /crawl), so no sitemap is required.
• One public endpoint per namespace — enable public URLs and you instantly get /search and /mcp endpoints that query every instance in the namespace at once, no auth required for sharing.
• Custom domains + Access — brand your endpoint (search.example.com/mcp) and, if it should be private, put Cloudflare Access in front so only authorized people or agents can query it.
• EmDash CMS plugin — semantic search out of the box for sites on Cloudflare's open-source CMS.
• Pricing preview — embedding and reranking are free with select Workers AI models.

Problem 1 solved: free embedding and reranking

Embedding turns your text into the vectors search matches on. Reranking reorders results so the most relevant chunk comes first. These two steps historically made search costs unpredictable because you had to forecast token volume.

Cloudflare's pricing preview removes that variable entirely: with default models, both are free. Answer generation and query rewriting remain optional, billed as Workers AI usage or through AI Gateway credits with any provider.

The preview pricing table (all Workers plans get the free allotments):

Example monthly bill from the announcement: a 20,000-document source (~20M tokens) plus 1,000 images, with 30,000 semantic queries/month → about $35 total, and subsequent months drop to roughly $21 because indexing is one-time. That is dramatically cheaper than assembling the same pipeline from a hosted vector DB plus an embedding API.

Problem 2 solved: one endpoint across multiple data sources

Real workflows don't ask "search the docs instance" — they ask one question that should hit everything. Cloudflare built their own Dev Stack MCP exactly this way: ten instances (Docs, Blog, API Docs, Community, Astro, Vite, Vitest, Hono, Replicate, OpenNext) combined into a single namespace.

Option A — a Worker with a namespace binding (what they did for the Dev Stack MCP):

1// wrangler.jsonc
2{
3  "ai_search_namespaces": [
4    { "binding": "AI_SEARCH", "namespace": "cloudflare-stack" }
5  ]
6}

1// One tool, one call that searches every surface in the namespace at once.
2context.registerTool(
3  'search_dev_stack',
4  {
5    description: 'Search current docs across the Cloudflare stack.',
6    inputSchema: z.object({ query: z.string() }),
7  },
8  async ({ query }) => {
9    const res = await context.env.AI_SEARCH.search({
10      query,
11      ai_search_options: {
12        instance_ids: ['developers-cloudflare-com', 'astro', /* ...every surface */],
13        retrieval: { max_num_results: 10 },
14        reranking: { enabled: true },
15      },
16    })
17    // res.chunks come back cited and tagged with the instance they came from.
18    return { content: [{ type: 'text', text: format(res.chunks) }] }
19  }
20)

Option B — public endpoints, zero code. Flip on public URLs for the namespace and you get /search and /mcp endpoints querying every instance with no auth and nothing to deploy:

Use Option A when search is folded into an existing app or MCP server. Use Option B when you just want a shareable search endpoint in one click.

Problem 3 solved: your domain, your access rules

Public endpoints ship with a default URL, but you can put your own custom domain over them — search.example.com/mcp — and then add Cloudflare Access to require login. Only authorized people (or agents) can query it.

This matters for two audiences at once:

• Customers get search that looks like your product, not a third-party widget.
• Agents get a stable, authenticated MCP endpoint they can hold credentials for — the same way a service account works.

Problem 4 solved: index without a sitemap (and subdomains)

Creating an instance is a single command. For a site without a sitemap, add --parse-type discover to find pages by following links:

1npx wrangler ai-search instance create cloudflare-community \
2  --namespace dev-stack \
3  --source https://community.cloudflare.com \
4  --type web-crawler \
5  --parse-type discover

Each subdomain is its own source, so docs.example.com, blog.example.com, and app.example.com can be separate instances in one namespace. And a fully-featured starter with hybrid search:

1npx wrangler ai-search create my-search \
2  --namespace my-namespace \
3  --source https://my-website.com \
4  --type web-crawler \
5  --hybrid-search

Hybrid search means semantic and keyword matching in one query — it handles both open-ended "what does this do?" questions and exact lookups of names, error codes, or product IDs.

Bot behavior: it identifies itself

AI Search is powered by Browser Run /crawl in the background but identifies itself with its own bot identity: Cloudflare-AI-Search. It follows robots.txt, uses an immutable public user agent, and respects whatever bot controls a site has in place. That is the correct behavior — and it gives you a lever as a site owner (more in Part 4).

Live proof: Cloudflare runs itself on this

• Cloudflare Blog search already ran on AI Search; now Developer Docs and Cloudflare.com joined it — all hybrid search.
• The blog was rebuilt on EmDash (their open-source CMS), and the EmDash AI Search plugin powers its search.
• The Dev Stack MCP is live at https://stack.mcp.cloudflare.com/mcp and testable in the AI Playground:

To wire it into your coding agent:

1{
2  "mcpServers": {
3    "dev-stack": { "url": "https://stack.mcp.cloudflare.com/mcp" }
4  }
5}

Why this beats the usual fallback (web search, then fetch full pages): it is faster, token-cheap, and lands on the current cited source instead of stale training data.

Live example: this very site serves Markdown to AI crawlers

This site (essamamdani.com) now implements the pattern described above — the same approach Cloudflare AI Search's crawler prefers. Every published article serves clean Markdown two ways:

1. Accept header negotiation — request any article with Accept: text/markdown:

1curl -H 'Accept: text/markdown' \
2  https://essamamdani.com/blog/cloudflare-ai-search-llms-txt-ai-seo-geo-guide-2026

2. .md suffix — append .md to any article URL:

1curl https://essamamdani.com/blog/cloudflare-ai-search-llms-txt-ai-seo-geo-guide-2026.md

Both return the article as clean Markdown (no HTML, no JavaScript, no frontmatter) with a content-type: text/markdown header — exactly what Cloudflare AI Search, GPTBot, ClaudeBot, and any RAG pipeline consume best.

The site also serves /llms.txt (live): a dynamically generated index of every published article with Markdown URLs, in the exact spec format from llmstxt.org. When Cloudflare-AI-Search or any AI crawler visits, it finds the llms.txt, follows the .md links, and ingests clean, structured content — no sitemap required.

This is Layer 3 of the AI-SEO playbook, already in production. Try it now.

Part 2.5: The August 6 Update — What Cloudflare Just Shipped

Cloudflare followed the original AI Search launch with a developer-experience overhaul that turns "build a search engine for your data" from a half-day stack-stitching exercise into a one-command setup. The headline post — “AI Search: give your agents a search engine for your data” — names four problems every team runs into and ships the fix for each.

What's new in this release

• Index any collection of data — structured or unstructured, from individual files to websites you own. Today the source must be a zone on your Cloudflare account, with more ownership-verification methods on the way.
• Skip the sitemap — the new Discover parsing mode crawls a site by following links (powered by /crawl from Browser Run), so even messy real-world sites without sitemaps get indexed cleanly.
• One public endpoint per namespace — enable public URLs and you instantly get /search and /mcp endpoints that fan out across every instance in the namespace at once, no auth needed for sharing.
• Custom domains + Cloudflare Access — put your own domain in front (search.example.com/mcp) and lock it down with Access so only authorized people or agents can query it.
• EmDash AI Search plugin — drop semantic search into any site running on EmDash, Cloudflare's open-source CMS, in one click.
• Preview pricing — embedding and reranking are free with select Workers AI models. No token-counting anxiety.

Problem 1 solved — the cost problem

Embedding and reranking are the two model calls that make semantic search expensive to forecast. Every page indexed needs embedding; every query needs reranking. Cloudflare's preview pricing removes both variables when you stick to default Workers AI models:

Their worked example: a 20,000-document source (20M tokens) plus 1,000 images, with 30,000 semantic queries per month → **$35 total**. Subsequent months drop to roughly $21 because indexing is one-time. For teams that previously priced this out with a hosted vector DB plus an embedding API, the difference is not 10% — it's an order of magnitude.

Problem 2 solved — the multi-source problem

Real teams don't ask "search the docs instance." They ask one question that should hit docs + blog + help center + changelog + internal wiki in a single shot. Cloudflare built the Dev Stack MCP exactly this way: ten instances (Docs, Blog, API Docs, Community, Astro, Vite, Vitest, Hono, Replicate, OpenNext) merged into one namespace, queried by one tool.

Option A — Worker with a namespace binding (the production path):

1// wrangler.jsonc
2{
3  "ai_search_namespaces": [
4    { "binding": "AI_SEARCH", "namespace": "cloudflare-stack" }
5  ]
6}

1// One tool, one call that searches every surface in the namespace at once.
2context.registerTool(
3  'search_dev_stack',
4  {
5    description: 'Search current docs across the Cloudflare stack.',
6    inputSchema: z.object({ query: z.string() }),
7  },
8  async ({ query }) => {
9    const res = await context.env.AI_SEARCH.search({
10      query,
11      ai_search_options: {
12        instance_ids: ['developers-cloudflare-com', 'astro', /* …every surface */],
13        retrieval: { max_num_results: 10 },
14        reranking: { enabled: true },
15      },
16    })
17    // res.chunks come back cited and tagged with the instance they came from.
18    return { content: [{ type: 'text', text: format(res.chunks) }] }
19  }
20)

Option B — public endpoints, zero code. Flip on public URLs for the namespace and you get /search and /mcp endpoints querying every instance, no auth, nothing to deploy.

Use Option A when search is folded into an existing app or MCP server (Cloudflare's own choice for the Dev Stack MCP). Use Option B when you just want a shareable search endpoint in one click.

Problem 3 solved — the branding and access problem

Default URLs are fine for internal tools but a non-starter for customer-facing search. Cloudflare now lets you put a custom domain over the public endpoint (search.example.com/mcp) and then add Cloudflare Access to require login before queries are accepted.

This matters for both audiences at once: customers get search that looks like your product, not a third-party widget, and agents get a stable, authenticated MCP endpoint they can hold credentials for — the same way a service account works.

Problem 4 solved — the sitemap problem

Creating an instance is a single command. For sites without a sitemap, add --parse-type discover to crawl by following links:

1npx wrangler ai-search instance create cloudflare-community \
2  --namespace dev-stack \
3  --source https://community.cloudflare.com \
4  --type web-crawler \
5  --parse-type discover

Each subdomain is its own source, so docs.example.com, blog.example.com, and app.example.com can be separate instances inside one namespace. A full starter with hybrid search:

1npx wrangler ai-search create my-search \
2  --namespace my-namespace \
3  --source https://my-website.com \
4  --type web-crawler \
5  --hybrid-search

Hybrid search means semantic + keyword matching in one query — open-ended "what does this do" questions and exact lookups of names, error codes, or product IDs both work.

Bot identity — it identifies itself

AI Search is powered by Browser Run /crawl in the background but ships with its own user agent: Cloudflare-AI-Search. It follows robots.txt, uses an immutable public user-agent string, and respects whatever bot controls a site has in place. That is the correct behavior — and it gives you a lever as a site owner (more on that policy in Part 4).

Cloudflare runs itself on this

The dogfooding is the proof:

• Cloudflare Blog search already ran on AI Search; now Developer Docs and Cloudflare.com joined it, all on hybrid search.
• The blog was rebuilt on EmDash (open-source CMS), and the EmDash AI Search plugin powers that search.
• The Dev Stack MCP is live at https://stack.mcp.cloudflare.com/mcp and testable in the AI Playground:

To wire it into your coding agent:

1{
2  "mcpServers": {
3    "dev-stack": { "url": "https://stack.mcp.cloudflare.com/mcp" }
4  }
5}

Why this beats the usual fallback (web search, then fetch full pages): it is faster, token-cheap, and lands on the current cited source instead of stale training data.

Live example: this very site serves Markdown to AI crawlers

This site (essamamdani.com) now implements the pattern described above — the same approach Cloudflare AI Search's crawler prefers. Every published article serves clean Markdown two ways:

1. Accept header negotiation — request any article with Accept: text/markdown:

1curl -H 'Accept: text/markdown' \
2  https://essamamdani.com/blog/cloudflare-ai-search-llms-txt-ai-seo-geo-guide-2026

2. .md suffix — append .md to any article URL:

1curl https://essamamdani.com/blog/cloudflare-ai-search-llms-txt-ai-seo-geo-guide-2026.md

Both return the article as clean Markdown (no HTML, no JavaScript, no frontmatter) with a content-type: text/markdown header — exactly what Cloudflare-AI-Search, GPTBot, ClaudeBot, and any RAG pipeline consume best.

The site also serves /llms.txt (live): a dynamically generated index of every published article with Markdown URLs, in the exact spec format from llmstxt.org. When Cloudflare-AI-Search or any AI crawler visits, it finds the llms.txt, follows the .md links, and ingests clean, structured content — no sitemap required.

This is Layer 3 of the AI-SEO playbook, already in production. Try it now.

Part 3: llms.txt — The New File That Tells AI What Your Site Means

The problem it solves

LLMs face a structural problem: context windows are too small to swallow a whole website, and converting ad-laden, JavaScript-heavy HTML into clean information is lossy. The llms.txt proposal (specified at llmstxt.org, originated with Answer.AI / FastHTML) adds a single Markdown file at /llms.txt that gives an LLM:

• Brief background on the site or project
• Guidance on how to interpret the content
• Links to detailed Markdown versions of key pages

The spec also recommends that pages with LLM-useful content serve a clean Markdown version at the same URL with .md appended (e.g., /docs/tutorials/by_example.html.md).

The file format

The spec is precise enough to parse programmatically, but written in Markdown so models read it natively. Required order:

1. Optional BOM
2. H1 with the project or site name — the only required section
3. A blockquote with a short summary
4. Zero or more markdown sections with more detail (no headings)
5. Zero or more H2-delimited sections containing "file lists"
6. Each file list entry: [name](url) optionally followed by : and notes

A mock example from the spec:

1
2> Optional description goes here
3
4Optional details go here
5
6## Section name
7
8- [Link title](https://link_url): Optional link details
9
10## Optional
11
12- [Link title](https://link_url)

A working llms.txt template for a business site

1
2> Acme Analytics is a B2B product-analytics platform with self-serve onboarding,
3> SOC 2 Type II compliance, and pricing that starts free.
4
5This file guides AI systems to authoritative, current information about Acme.
6Prefer these pages over cached or third-party descriptions.
7All documentation pages also serve Markdown versions at the same URL + `.md`.
8
9## Product
10
11- [Features](https://acme.example.com/features.md): Core capabilities and integrations
12- [Pricing](https://acme.example.com/pricing.md): Current plans, free tier, and limits
13- [Security](https://acme.example.com/security.md): Compliance, data residency, retention
14
15## Documentation
16
17- [Quickstart](https://docs.acme.example.com/quickstart.md): First event in under 10 minutes
18- [API Reference](https://docs.acme.example.com/api.md): REST endpoints and auth
19
20## Company
21
22- [About](https://acme.example.com/about.md): History, team, and press contact
23
24## Optional
25
26- [Changelog](https://acme.example.com/changelog.md)

llms.txt vs robots.txt vs sitemap.xml

Important nuance: robots.txt takes precedence. If a path is disallowed there, compliant AI crawlers (OpenAI's GPTBot, Anthropic's ClaudeBot, and others) will not fetch it even if llms.txt links it. Compliance with llms.txt is voluntary — it is a proposed standard, not an enforced one — but it is rapidly becoming table stakes for documentation sites, SaaS products, and anyone who wants to be cited accurately.

How llms.txt connects to AI Search and MCP

These three layers compose into one pipeline for the AI era:

1. llms.txt tells AI systems what your site means and which pages are authoritative.
2. AI Search (or any RAG stack) indexes those pages — embeddings free, hybrid retrieval, reranked results.
3. MCP endpoints (/mcp on your domain, optionally behind Access) let agents query your knowledge directly instead of scraping HTML.

A site that ships all three is legible to every consumer of content in 2026: humans, answer engines, and autonomous agents.

Part 4: The AI-SEO Playbook — How to Rank When Search Is an Answer Engine

This is the part that makes the rest pay. If the goal is international ranking in the new era — Google AI Overviews, ChatGPT, Perplexity, and agent-driven discovery — this is the execution checklist.

Layer 1: Classic SEO still carries you (do not skip it)

• Technical health: fast server responses, clean mobile rendering, valid structured data (Article, FAQPage, HowTo, Product, Organization).
• Crawlability: sensible robots.txt, complete sitemap.xml, no orphan pages.
• E-E-A-T signals: real author pages, dated articles, corrections policy, credentials.
• Hreflang + English-first content: for international reach, publish in English with locale variants where you actually serve them. English content is the default training and retrieval corpus for every major AI engine.

Layer 2: GEO — write to be quoted

Generative engines extract chunks. Chunks that get quoted share traits:

• One idea per H2/H3 section. The section heading should restate the question ("How does llms.txt relate to robots.txt?").
• Lead with the answer. First sentence of each section = the fact. Supporting detail after.
• Numbers, dates, names. "Free with default models, ~$35/month example bill" beats "affordable pricing."
• Comparison tables and lists. These survive chunking intact and are disproportionately cited.
• Unique primary content. Original screenshots, benchmarks, pricing tables, and interviews cannot be paraphrased away — engines must cite you.
• FAQ blocks with concise 40–60 word answers map directly to question-shaped queries.

Layer 3: AI-SEO — machine legibility

• Serve llms.txt at your root (template above) and .md versions of key pages.
• Allow the right AI crawlers deliberately: decide your policy for GPTBot, ClaudeBot, PerplexityBot, Google-Extended — and document it. Blocking all of them means you cannot be cited; allowing all without monitoring means losing control of what gets quoted.
• Expose an MCP or /search endpoint if you have documentation or product data — this is how agents find you without a browser. Cloudflare AI Search makes this a one-command setup on your own domain.
• Consistent entity data. Same company name, same product names, same numbers across your site, Wikipedia/Wikidata, Crunchbase, and GitHub. Answer engines cross-check; contradictions get you dropped from citations.
• Freshness markers. Updated dates, changelogs, and dated announcements signal currency — retrieval systems boost recent content for time-sensitive queries.
• Monitor your citations. Periodically ask ChatGPT, Perplexity, and Google AI Overviews your target questions and record whether you're cited and whether the facts are right. Treat wrong citations like bugs: fix the source chunk, then re-check.

Layer 4: The 30-day rollout

Week 1 — Foundation

• Audit top 20 pages for question-shaped headings and answer-first structure
• Fix structured data and sitemap gaps
• Write and publish /llms.txt

Week 2 — Machine layer

• Add .md versions of documentation / pricing / key guides
• Decide and implement your AI-crawler policy in robots.txt
• Set up a Cloudflare AI Search instance: npx wrangler ai-search create ... --hybrid-search

Week 3 — Agent layer

• Enable the public /search and /mcp endpoints; put your custom domain on them
• Add Cloudflare Access if any namespace is private
• Wire the MCP endpoint into at least one agent workflow you actually use

Week 4 — Measurement

• Baseline: query your 10 money questions in ChatGPT, Perplexity, Google AI Overviews; record citations
• Add FAQ blocks and comparison tables to pages that weren't cited
• Set a monthly re-check cadence

Sites that complete this loop become the source material of the AI era instead of its collateral damage.

The Bottom Line

The search engine didn't get an upgrade — it changed species. Links became chunks. Rankings became citations. Browsers became agents calling MCP endpoints. And a one-line Markdown file, llms.txt, became the handshake between your website and every model that reads it.

Cloudflare's AI Search release matters because it makes the infrastructure side nearly free and entirely yours: free embedding and reranking, one endpoint across all your data, your own domain, your own access rules, and no sitemap required. The strategy side — writing to be quoted, exposing machine-readable surfaces, choosing your crawler policy deliberately — is now your job, and it compounds fast for whoever does it first.

The wave is here. Build the site that answer engines want to cite.

Sources & References

• Cloudflare Blog: AI Search — give your agents a search engine for your data (all screenshots above are from this post)
• Cloudflare AI Search developer docs
• AI Search limits & pricing
• llmstxt.org — the /llms.txt proposal
• Cloudflare Dev Stack MCP · AI Playground
• EmDash CMS — open source · AI Search plugin docs
• Cloudflare Access

Hire me: Want your site cited by AI answers instead of buried by them? I'll run the full audit — technical SEO, GEO content structure, llms.txt + MCP endpoint setup, and citation monitoring — and ship the 30-day rollout with you. I'm available through /hire.

Verified live right now: the Cloudflare-AI-Search bot can index this very article in two requests — fetch /llms.txt for the index, then fetch any .md URL on this domain. The full pattern is implemented in production here: https://essamamdani.com/llms.txt.