Use Markdown for Agents for all docs markdown generation (#28921)

* first pass refactor

* handle redirect in code, not cloudflare rules

* use style guide as example for markdown pages

* removed application handling on index.md

* update llms.txt to not use index.md by default

* copy edits

* copy edit

* llms.txt improvements

* dont use MFA abbreviation

* fix llms.txt spacing issue

* add back removed test

* Use correct url for llms.txt and markdown files

* remove duplicate llms.txt entries

* use index.md links in llms.txt

* llms.txt block quote updates

* use a better example url in ai tooling page

* add index.md to starlightLinksValidator exclude

* Update astro.config.ts

Co-authored-by: ask-bonk[bot] <249159057+ask-bonk[bot]@users.noreply.github.com>

* Update src/pages/llms.txt.ts

Co-authored-by: ask-bonk[bot] <249159057+ask-bonk[bot]@users.noreply.github.com>

* Fix llms.txt section labels sometimes being incorrect

---------

Co-authored-by: ask-bonk[bot] <249159057+ask-bonk[bot]@users.noreply.github.com>

Author: mvvmm

astro.config.ts

@@ -145,6 +145,7 @@ export default defineConfig({
 									"/llms.txt",
 									"/llms-full.txt",
 									"**/llms.txt",
+									"**/index.md",
 									"{props.*}",
 									"/",
 									"/glossary/",
@@ -156,10 +157,8 @@ export default defineConfig({
 									"/workers/examples/?tags=*",
 									"/workers/llms-full.txt",
 									"/workers-ai/models/**",
-									"**index.md",
 									"/markdown.zip",
 									"/style-guide/index.md",
-									"/style-guide/fixtures/markdown/index.md",
 									"/videos/**",
 									"/search/**",
 								],

src/content/docs/style-guide/fixtures/index.mdx

@@ -41,13 +41,13 @@ The references to the panels `id`, usually handled by JavaScript, are visible bu
 
 ### Turning our components into "Markdownable" HTML
 
-To solve this, we created a [`rehype plugin`](https://github.com/cloudflare/cloudflare-docs/blob/d5a19deded110bce6a7c5d45e702d36527da0a4e/src/plugins/rehype/filter-elements.ts) for:
+To solve this, we use [Markdown for Agents](/fundamentals/reference/markdown-for-agents/), which converts HTML to Markdown at the Cloudflare network layer. It handles:
 
-- Removing non-content tags (`script`, `style`, `link`, etc) via a [tags allowlist](https://github.com/cloudflare/cloudflare-docs/blob/d5a19deded110bce6a7c5d45e702d36527da0a4e/src/plugins/rehype/filter-elements.ts#L19-L104)
-- [Transforming custom elements](https://github.com/cloudflare/cloudflare-docs/blob/d5a19deded110bce6a7c5d45e702d36527da0a4e/src/plugins/rehype/filter-elements.ts#L189-L227) like `starlight-tabs` into standard unordered lists
-- [Adapting our Expressive Code codeblocks HTML](https://github.com/cloudflare/cloudflare-docs/blob/d5a19deded110bce6a7c5d45e702d36527da0a4e/src/plugins/rehype/filter-elements.ts#L143-L178) to the [HTML that CommonMark expects](https://spec.commonmark.org/0.31.2/#example-142)
+- Removing non-content tags (`script`, `style`, `link`, etc.)
+- Transforming custom elements like `starlight-tabs` into standard unordered lists
+- Adapting code block HTML into clean Markdown fenced code blocks
 
-Taking the `Tabs` example from the previous section and running it through our plugin will now give us a normal unordered list with the content properly associated with a given list item:
+Taking the `Tabs` example from the previous section, Markdown for Agents will give us a normal unordered list with the content properly associated with a given list item:
 
 ```md
 - One
@@ -59,20 +59,25 @@ Taking the `Tabs` example from the previous section and running it through our p
   Two Content
 ```
 
-For example, take a look at our Markdown test fixture (or any page by appending `/index.md` to the URL):
+You can request any page as Markdown in two ways:
 
-- [`/style-guide/fixtures/markdown/`](/style-guide/fixtures/markdown/)
-- [`/style-guide/fixtures/markdown/index.md`](/style-guide/fixtures/markdown/index.md)
+- Send a request with an `Accept: text/markdown` header:
+
+  ```bash
+  curl "https://developers.cloudflare.com/style-guide/ai-tooling/" \
+    --header "Accept: text/markdown"
+  ```
+
+- Append `index.md` to the URL — for example, [`/style-guide/ai-tooling/index.md`](/style-guide/ai-tooling/index.md)
 
 ### Saving on tokens
 
-Most AI pricing is around input & output tokens and our approach greatly reduces the amount of input tokens required.
+Most AI pricing is around input & output tokens and Markdown greatly reduces the amount of input tokens required.
 
 For example, let's take a look at the amount of tokens required for the [Workers Get Started](/workers/get-started/guide/) using [OpenAI's tokenizer](https://platform.openai.com/tokenizer):
 
 - HTML: 15,229 tokens
-- turndown: 3,401 tokens (4.48x less than HTML)
-- index.md: 2,110 tokens (7.22x less than HTML)
+- Markdown: 2,110 tokens (7.22x less than HTML)
 
 When providing our content to AI, we can see a real-world ~7x saving in input tokens cost.
 

src/content/docs/style-guide/fixtures/markdown.mdx

@@ -11,15 +11,17 @@ We have implemented `llms.txt` and `llms-full.txt` as follows:
 
 To obtain a Markdown version of a single documentation page, you can:
 
-- Send a request to `/$page/index.md` — Add `/index.md` to the end of any page to get the Markdown version. For example, [`/style-guide/index.md`](/style-guide/index.md).
-
-- Send a request to `/$page/` with an `Accept` header requesting a page in Markdown format — Uses [Markdown for Agents](/fundamentals/reference/markdown-for-agents/). For example:
+- Send a request to any page with an `Accept: text/markdown` header — Uses [Markdown for Agents](/fundamentals/reference/markdown-for-agents/) to convert the page to Markdown at the network layer. For example:
 
   ```bash
-  curl "https://developers.cloudflare.com/style-guide/" \
+  curl "https://developers.cloudflare.com/style-guide/ai-tooling/" \
     --header "Accept: text/markdown"
   ```
 
+- Send a request to `/$page/index.md` — Add `/index.md` to the end of any page to get the Markdown version. For example, [`/style-guide/ai-tooling/index.md`](/style-guide/ai-tooling/index.md).
+
+Both methods return the same Markdown output, powered by [Markdown for Agents](/fundamentals/reference/markdown-for-agents/).
+
 In the top right of this page, you will see a `Page options` button where you can copy the current page as Markdown that can be given to your LLM of choice.
 
 <Width size="medium">

src/content/docs/style-guide/how-we-docs/ai-consumability.mdx

@@ -1,5 +1,4 @@
 import { defineMiddleware } from "astro:middleware";
-import { htmlToMarkdown } from "~/util/markdown";
 
 // `astro dev` only middleware so that `/api/...` links can be viewed.
 export const onRequest = defineMiddleware(async (context, next) => {
@@ -14,24 +13,6 @@ export const onRequest = defineMiddleware(async (context, next) => {
 					"accept-encoding": "identity",
 				},
 			});
-		} else if (
-			pathname.endsWith("/index.md") ||
-			context.request.headers.get("accept")?.includes("text/markdown")
-		) {
-			const htmlUrl = new URL(pathname.replace("index.md", ""), context.url);
-			const html = await (await fetch(htmlUrl)).text();
-
-			const markdown = await htmlToMarkdown(html, context.url.toString());
-
-			if (!markdown) {
-				return new Response("Not Found", { status: 404 });
-			}
-
-			return new Response(markdown, {
-				headers: {
-					"content-type": "text/markdown; charset=utf-8",
-				},
-			});
 		}
 	}
 

src/content/partials/style-guide/llms-txt.mdx

@@ -0,0 +1,166 @@
+import type { APIRoute, GetStaticPaths, InferGetStaticPropsType } from "astro";
+import { getCollection } from "astro:content";
+import dedent from "dedent";
+
+export const getStaticPaths = (async () => {
+	const directory = await getCollection("directory");
+
+	const docs = await getCollection("docs");
+
+	return directory
+		.map((entry) => {
+			const productUrl = entry.data.entry.url;
+			// Derive route segments from the product's canonical URL.
+			// e.g. /cloudflare-for-platforms/cloudflare-for-saas/ → ["cloudflare-for-platforms", "cloudflare-for-saas"]
+			// e.g. /workers/ → ["workers"]
+			// Skip the root URL "/" (home entry) and fragment-only anchors (e.g. /path/#anchor)
+			if (!productUrl || productUrl === "/" || productUrl.includes("#"))
+				return null;
+
+			const urlPath = productUrl.slice(1, -1); // strip leading/trailing slashes
+			if (!urlPath) return null;
+
+			const prefix = urlPath;
+			const pages = docs.filter(
+				(e) => e.id.startsWith(prefix + "/") || e.id === prefix,
+			);
+
+			if (pages.length === 0) return null;
+
+			return {
+				params: { product: urlPath },
+				props: { entry, pages },
+			};
+		})
+		.filter((p) => p !== null);
+}) satisfies GetStaticPaths;
+
+type Props = InferGetStaticPropsType<typeof getStaticPaths>;
+
+type Page = InferGetStaticPropsType<typeof getStaticPaths>["pages"][number];
+
+function formatPage(base: string, e: Page) {
+	const line = `- [${e.data.title}](${base}/${e.id}/index.md)`;
+	return e.data.description ? line.concat(`: ${e.data.description}`) : line;
+}
+
+interface Section {
+	label: string;
+	order: number | undefined;
+	indexPage: Page | undefined;
+	children: Page[];
+}
+
+function buildSections(prefix: string, pages: Page[]): Section[] | null {
+	const childPages = pages.filter((e) => e.id !== prefix);
+
+	// Find section index pages: pages that are exactly one directory level
+	// below the prefix and have children under them.
+	// e.g., for prefix "workers", find "workers/get-started", "workers/configuration", etc.
+	const sectionMap = new Map<string, Section>();
+
+	for (const page of childPages) {
+		const relative = page.id.slice(prefix.length + 1); // e.g., "get-started" or "get-started/guide"
+		const firstSegment = relative.split("/")[0];
+		const sectionId = `${prefix}/${firstSegment}`;
+
+		if (!sectionMap.has(sectionId)) {
+			sectionMap.set(sectionId, {
+				label: firstSegment,
+				order: undefined,
+				indexPage: undefined,
+				children: [],
+			});
+		}
+
+		const section = sectionMap.get(sectionId)!;
+
+		if (page.id === sectionId) {
+			// This is the section index page
+			section.indexPage = page;
+			section.label = page.data.title;
+			section.order = page.data.sidebar?.order;
+		} else {
+			section.children.push(page);
+		}
+	}
+
+	const sections = [...sectionMap.values()];
+
+	// Check if any sections have explicit sidebar ordering
+	const hasOrdering = sections.some((s) => s.order !== undefined);
+	if (!hasOrdering) return null;
+
+	// Sort sections: those with order first (by order), then those without (alphabetically by label)
+	sections.sort((a, b) => {
+		if (a.order !== undefined && b.order !== undefined)
+			return a.order - b.order;
+		if (a.order !== undefined) return -1;
+		if (b.order !== undefined) return 1;
+		return a.label.localeCompare(b.label);
+	});
+
+	return sections;
+}
+
+export const GET: APIRoute<Props> = async ({ props, url }) => {
+	const base = url.origin;
+	const { entry, pages } = props;
+	const { title, url: productUrl } = entry.data.entry;
+	const description = entry.data.meta?.description;
+
+	const prefix = productUrl.slice(1, -1);
+	const rootPage = pages.find((e) => e.id === prefix);
+	const rootLink = rootPage
+		? formatPage(base, rootPage)
+		: `- [${title}](${base}${productUrl}index.md)`;
+
+	const sections = buildSections(prefix, pages);
+
+	let pageContent: string;
+
+	if (sections) {
+		// Grouped output with section headers
+		pageContent = sections
+			.map((section) => {
+				const heading = `## ${section.label}`;
+				const lines: string[] = [];
+				if (section.indexPage) {
+					lines.push(formatPage(base, section.indexPage));
+				}
+				for (const child of section.children) {
+					lines.push(formatPage(base, child));
+				}
+				return `${heading}\n\n${lines.join("\n")}`;
+			})
+			.join("\n\n");
+	} else {
+		// Flat fallback
+		const childPages = pages.filter((e) => e.id !== prefix);
+		pageContent = childPages.map((e) => formatPage(base, e)).join("\n");
+	}
+
+	const pagesSection = sections
+		? `## Overview\n\n${rootLink}\n\n${pageContent}`
+		: `## ${title} documentation pages\n\n${rootLink}\n\n${pageContent}`;
+
+	const markdown = dedent(`
+		# ${title}
+
+		${description ?? ""}
+
+		> Links below point directly to Markdown versions of each page. Any page can also be retrieved as Markdown by sending an \`Accept: text/markdown\` header to the page's URL without the \`index.md\` suffix (for example, \`curl -H "Accept: text/markdown" ${base}${productUrl}\`).
+		>
+		> For other Cloudflare products, see the [Cloudflare documentation directory](${base}/llms.txt).
+		>
+		> Use [${title} llms-full.txt](${base}${productUrl}llms-full.txt) for the complete ${title} documentation in a single file, intended for offline indexing, bulk vectorization, or large-context models.
+
+		${pagesSection}
+	`);
+
+	return new Response(markdown, {
+		headers: {
+			"content-type": "text/plain",
+		},
+	});
+};