Make Your Website Readable by AI Agents: A Practical Checklist

More and more of your visitors are not people. AI assistants summarise pages, answer questions from them, and increasingly act on them. A site that is clear to a screen reader and a search crawler is most of the way to being clear to an agent too. The rest is a short checklist.

One honest caveat first: nobody outside the vendors can promise exactly how each assistant ranks or cites pages. Treat everything below as removing ambiguity, not as a guaranteed ranking trick.

1. Put the content in the HTML#

An agent that fetches your URL often does not run your JavaScript. If the text only appears after a client-side render, it may see an empty shell.

• Render content on the server or at build time. In Next.js, server components and static generation do this by default.
• Check what a bot sees: curl -s https://your-site.com/page | grep "a sentence from the page". If it is missing, so is your content.

2. Use semantic HTML#

Elements carry meaning that a div cannot:

• One <h1> per page, then <h2> and <h3> in order, without skipping levels.
• <nav>, <main>, <article>, <header>, <footer> for the page's regions.
• <time datetime="2026-10-26"> for dates, <ul> and <table> for real lists and tables.
• Descriptive link text. "Read the case study" says more than "click here".

This is plain accessibility work and it pays off twice.

3. State facts explicitly with structured data#

JSON-LD lets you say what a page is, in a format machines read without guessing:

<script type="application/ld+json">
{
  "@context": "https://schema.org",
  "@type": "BlogPosting",
  "headline": "Make Your Website Readable by AI Agents",
  "datePublished": "2026-10-26",
  "author": { "@type": "Person", "name": "Your Name", "url": "https://example.com" }
}
</script>

Use stable @id values so every page points at the same Person or Organization entity instead of redefining it. Validate with a schema checker, and make sure the markup matches what is visible on the page.

4. Consider an llms.txt#

llms.txt is a proposal, not a standard that every assistant honours: a plain Markdown file at /llms.txt that summarises the site and links to its key pages, written for a language model rather than a browser.

# Your Name - Software developer

> One or two sentences on who you are and what the site contains.

## Key pages
- [Projects](https://example.com/projects): case studies of shipped work
- [Blog](https://example.com/blog): technical write-ups

It costs twenty minutes, it forces you to write your own summary, and some tools do use it. Do not expect it to move your search ranking.

5. Decide your crawler policy on purpose#

robots.txt is where you say which bots may fetch what. Many AI companies publish separate user agents for training and for fetching a page on a user's request, and you can treat them differently.

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

Whatever you choose, choose it deliberately, and keep a current sitemap.xml so crawlers find new posts without guessing. Remember that robots.txt is a request, not access control. Anything private needs real authentication.

6. Make titles and descriptions do real work#

An agent often meets your page as a title, a description and a snippet before it reads anything else.

• Titles: specific, around 60 characters or fewer, with the topic first.
• Meta descriptions: one honest sentence of 120 to 160 characters that says what the reader will get.
• A canonical URL on every page, so duplicates collapse into one.

7. Keep pages fast and stable#

Slow or flaky pages get fetched less and abandoned more, by people and bots alike. Keep Core Web Vitals healthy, serve proper status codes (a real 404 for a missing page, not a 200 with an error message), and avoid content hidden behind interactions an agent will never perform.

8. Watch the emerging part#

Proposals such as WebMCP aim to let a site expose actions, like "book a slot" or "search the catalogue", as tools an agent can call directly instead of scraping buttons. It is early, and not something to build a plan around today, but it points the same direction as everything above: be explicit about what your site is and what it can do.

The short version#

• Content is in the server-rendered HTML
• Semantic headings and landmarks
• JSON-LD with stable entity ids
• A written llms.txt
• A deliberate robots.txt and an up-to-date sitemap
• Specific titles, honest descriptions, canonical URLs
• Fast pages with correct status codes