Skip to main content

Is your website ready for AI search? A practical checklist for UK businesses

Your website is ready for AI search when it is indexed, open to the right crawlers, easy to read as plain text, and answers real questions clearly. The checklist we ran on our own site.

Abstract illustration of a robots.txt document lying on a desk beside a search bar and two glass browser windows.

Your website is ready for AI search when it is indexed, open to the right crawlers, easy to read as plain text, and answers real questions clearly. There is no secret file or special markup. Google says so directly, and most of the work is ordinary SEO done properly. Below is the checklist we ran on mtxstudio.com, with what we actually changed.

What "AI search" means for a business site

When someone asks ChatGPT, Google's AI Overviews or Perplexity which web studio, accountant or plumber to use, the answer is built from web pages the system can find and trust. Two things decide whether your site is one of them:

  1. Can the system reach and read your pages?
  2. Do your pages give a clear, specific answer it can cite?

Google's guidance for AI Overviews and AI Mode is blunt: "There are no additional requirements to appear in AI Overviews or AI Mode, nor other special optimizations necessary." A page has to be indexed and eligible to show with a snippet, and that is it (Google Search Central, checked 9 November 2026).

So the checklist below is mostly about not getting in your own way.

The checklist

#CheckWhy it mattersHow to check it
1Your important pages are indexedGoogle only uses indexed pages in AI featuresSearch Console, Pages report and URL Inspection
2robots.txt allows the crawlers you wantBlocking a crawler keeps you out of that product's answersOpen yoursite.co.uk/robots.txt
3Your firewall or CDN isn't blocking themBot protection can stop crawlers that robots.txt allowsYour hosting or CDN firewall logs
4Key content is in plain HTML textText inside images, sliders or scripts is hard to readView the page source, or turn JavaScript off
5Each page answers one question firstAnswers are easier to quote when they come firstRead the first two sentences of each page
6Real names, a real address and real proofSystems and people both trust specificsAbout, contact and case study pages
7Structured data matches the visible pageMismatched schema is ignored or worseGoogle's Rich Results Test
8Pages link to each other sensiblyInternal links help crawlers find and understand pagesA crawl, or just click through your own site
9You measure itOtherwise you are guessingSearch Console and GA4

1. Indexing comes first

If a page isn't indexed, nothing else on this list matters for Google. When we relaunched mtxstudio.com in October, a lot of our pages were "Discovered, currently not indexed". Better internal links, stronger case study pages and indexing requests in Search Console moved most of them in. A few new posts are still waiting, which is normal for a young page.

2. Decide which AI crawlers you allow

Each AI company uses its own crawler names, and some separate search from training. OpenAI, for example, uses OAI-SearchBot to show sites in ChatGPT search and GPTBot for training, and you can allow one without the other (OpenAI, checked 9 November 2026).

We decided we want to be found, so our robots.txt allows the main search and AI crawlers by name. This is the pattern, repeated for each one:

User-Agent: OAI-SearchBot
Allow: /
Disallow: /admin
Disallow: /api
Disallow: /private
Disallow: /studio

On our site the same block covers GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot, Google-Extended, Applebot-Extended and Bingbot, plus a link to our sitemap. If you'd rather not have your content used for training, you can allow the search crawlers and disallow the training ones. That is a business decision, not a technical one.

3. Check your firewall as well as robots.txt

This is the one most people miss. Google's own list of best practices says crawling should be allowed "in robots.txt, and by any CDN or hosting infrastructure".

Our site runs on Vercel with bot protection switched on, because it cuts out a lot of junk traffic. Verified crawlers such as Googlebot pass, and we confirmed that in Search Console. Unverified tools get a challenge page instead of the site. That is the trade we chose, and it is worth checking what your own host or security plugin does. A WordPress security plugin or a Cloudflare rule can block a crawler without anyone noticing.

4. Keep the important words in text

AI systems read text. If your services, contact details or case study results only appear inside images, carousels or PDFs, they are easy to miss. Our pages are server-rendered, so the full text is in the HTML before any script runs. On WordPress that is the default, as long as the key content isn't buried in sliders or image text.

5. Answer first, then explain

Every post we publish now opens with the answer in the first two or three sentences, then explains it, then ends with a short FAQ. This post does the same. It makes pages easier to quote, and it is simply better for readers in a hurry. If someone asks "is SiteGround good for WordPress?" the best page to cite is the one that says yes or no, and why, at the top.

6. Be specific about who you are

Vague pages are hard to trust. Ours name the people who do the work, state where the company is registered, and list the projects we've actually built. Each post has a named author with a short bio. If your site says "a team of experts" and nothing else, that is the first thing to fix.

7. Structured data that matches the page

Structured data helps machines understand what a page is about, but it has to match what visitors see. Google says there is "no special schema.org structured data" needed for AI features, so don't add markup for its own sake. On our site the useful pieces are:

  • Organisation details as a professional service, with the same name, address and contact details as the visible pages
  • The founders as named people, linked from the posts they write
  • Articles with a real author and real dates
  • FAQ markup only where the page shows those questions and answers

Crawlers find pages by following links, and links tell them how pages relate. Early this month we added a planned set of internal links across our service pages, posts and case studies, then checked every one on the live site. It is unglamorous work, and it is still one of the most useful things you can do for a small site.

9. Measure it

Search Console now includes AI Overviews and AI Mode traffic inside the normal Performance report, under the "Web" search type. GA4 can also show visits that arrive from AI assistants as their own channel. For most small businesses these numbers start tiny. That's fine. What matters is that they're tracked from the start, so you can see whether the work is paying off.

What about llms.txt?

llms.txt is a proposed file that summarises your site for language models. We publish one, generated automatically from our content, so it lists our services, key pages, posts and projects and stays up to date. It took little effort, so we kept it.

Be realistic about it, though. Google says you don't need "new machine readable files, AI text files, or markup" to appear in its AI features. Treat llms.txt as optional housekeeping, not a ranking trick, and don't pay anyone to "optimise" it.

What we would not do

  • Mass-produce thin pages for every town or question
  • Add schema that doesn't match the visible page
  • Stuff pages with keywords or fake FAQs
  • Block every bot by default, then wonder why nothing finds you

If you'd like a quick first look, run our Free Website Scan. It checks the basics on this list and emails you what we'd fix first.

Questions

Do I need a special file or plugin to appear in AI search?

No. Google says there are no additional requirements for its AI features beyond being indexed and eligible for a snippet. An llms.txt file is optional.

Should I block AI crawlers?

It depends on what you want. If you want to be found in AI answers, allow the search crawlers. You can block training crawlers such as GPTBot separately if you prefer.

Does AI search replace normal SEO?

No. The same fundamentals apply: indexing, crawlable pages, clear content, internal links and a good page experience.

How do I know if AI search is sending me visitors?

Search Console includes AI Overviews and AI Mode in the Performance report. GA4 can show AI assistant referrals as their own channel.

Is this different for WordPress sites?

Not much. WordPress outputs plain HTML by default. The usual problems are a security plugin blocking crawlers, key content hidden in sliders, and pages that never got indexed.

Liked this? Get the next one in your inbox.

Seen enough thinking? Let's do some doing.