How this site serves Markdown to agents

An AI agent that wants to know what a company does fetches its homepage and gets a document built for a browser: navigation markup, hydration payloads, an inlined logo, a font preload, a consent island. The sentences it actually came for are in there somewhere. Everything else is cost — tokens spent parsing chrome, plus a real chance the agent quotes a menu label back as if it were a claim the company made.

The fix is old and boring: content negotiation. If a client says it would rather have text/markdown, give it Markdown. This is the build log for how that works here, written mostly so the next person changing it knows which lines are load-bearing.

Ask for Markdown, get Markdown#

The negotiation happens in one place, middleware.ts, before routing:

export default function middleware(req: NextRequest) {
  const { pathname } = req.nextUrl
 
  if (acceptsMarkdown(req.headers) && pathIsNegotiable(pathname)) {
    const url = req.nextUrl.clone()
    url.pathname = pathname === "/" ? "/api/md" : `/api/md${pathname}`
    const res = NextResponse.rewrite(url)
    res.headers.set("vary", "accept")
    return res
  }
  // …locale routing for everything else
}

Two predicates and a rewrite. acceptsMarkdown reads the request's Accept header. pathIsNegotiable decides whether this URL has a Markdown answer at all. When both hold, the request is rewritten — not redirected — to /api/md/<the same path>, so the URL an agent cites is the URL a human opens. One address, two representations.

vary: accept is the load-bearing line. Without it, a CDN that cached the Markdown for one request would cheerfully hand it to the next browser that asked for HTML.

curl -H "Accept: text/markdown" https://anby.ai/blog/how-this-site-serves-markdown-to-agents

Not every path is negotiable#

The allow-list lives in lib/agent-readiness/middleware-predicates.ts. Blog URLs are matched by shape rather than enumerated, because posts arrive as files and nobody should have to remember to register one:

const BLOG_NEGOTIABLE = [
  /^\/blog$/,
  /^\/blog\/(?!rss\.xml$)[^/]+$/,
  /^\/(en|vi)\/blog\/(?!rss\.xml$)[^/]+$/,
]
 
export function acceptsMarkdown(headers: {
  get(name: string): string | null
}): boolean {
  if (headers.get("x-md-bypass") === "1") return false
  const accept = headers.get("accept") || ""
  return accept.toLowerCase().includes("text/markdown")
}

Three details in there are deliberate.

The negative lookahead keeps rss.xml out. It sits under /blog/ but it is a feed route, not a slug, and converting it would produce nonsense.

Retired routes are absent from the list entirely. Several old paths now answer with a redirect; serving Markdown for a redirect would tell an agent the page still exists, which is the opposite of what the redirect is for.

And x-md-bypass exists because of a loop. For an ordinary HTML page the converter has nothing to read from disk, so it fetches the page back from its own origin and converts the response. Without the bypass header on that internal fetch, the middleware would negotiate with itself, forever.

Two sources, one output#

That gives the route two jobs, and they are not equally good.

For a rendered page, /api/md fetches the HTML with the bypass header, drops the script, style, nav, footer and svg subtrees, then runs Turndown over what is left. It is a translation, and translations lose things.

For a blog post there is no translation at all, because the source was already Markdown before it was ever a page. The route reads the same .mdx file the page component reads:

export async function convertBlogPostToMarkdown(args: {
  locale: Locale
  slug: string
}): Promise<string | null> {
  if (args.locale !== "en" && args.locale !== "vi") return null
  const post = await loadBlogPost({ slug: args.slug, locale: args.locale })
  if (!post || post.frontmatter.draft) return null
  const cleaned = stripJsxComponents(post.body).trim()
  const parts = [`# ${post.frontmatter.title}`, post.frontmatter.date, cleaned]
  return parts.join("\n\n") + "\n"
}

Frontmatter comes off, a # title and a date line go on, and the body is handed over close to as-authored. That is why the blog is the only MDX path still wired to this route: it is the one corpus where the Markdown an agent receives and the page a reader sees are the same text, not two renderings that can drift.

Where it degrades#

Three honest limitations, all of them visible in the code above.

Components flatten, and one disappears. stripJsxComponents is a set of regexes, not a compiler. A block component keeps its inner Markdown and loses its wrapper — the callout you just read arrives as bare paragraphs, its title gone. A self-closing component is deleted outright, which means a <Figure /> takes its caption and alt text with it. A plain ![alt](/path.png) survives intact, so an image that matters to the argument is better written as Markdown than as a component.

Expressions are blocked by design. Component attributes are quoted strings — width="1200", never width={1200}. A post is data, not code, and a corpus an agent can read should not contain anything that has to be evaluated to be understood.

Only title and date survive from the frontmatter. The description and tags are page metadata; they never reach the Markdown. An agent reading this file gets the argument, not the taxonomy.

None of that is fixed by making the regexes cleverer. It is fixed by writing posts whose meaning lives in the prose, which is a better constraint than it sounds.

This post is the test case. It was written as a draft specifically to run the whole path once — file on disk, index card, share image, feed entry, Markdown response — before any of it carried something worth reading. When it leaves draft, the curl above returns it. Until then, that command returns a 404, which is also the correct answer.