ResilientNiche

Technical Setup signal

A foundation: pass or fail

Can AI actually read your site?

Before an AI engine can quote you, its search crawler has to reach your pages and be allowed to read them. This is one of the two foundations: a short checklist that can be finished, and once every check passes it is done.

Technical Setup

Reading your site…

  • AI search crawlers can fetch your pages: no robots.txt rule blocks OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User or PerplexityBot
  • None of your pages are hidden from search: no noindex tag on the pages we read
  • You have a sitemap, so an engine can find pages nothing links to
  • Every page tells an engine what kind of page it is: schema markup on each page we read

Why it matters

What an engine does with it

ChatGPT, Claude and Perplexity fetch pages with their own search crawlers: OAI-SearchBot and ChatGPT-User, Claude-SearchBot and Claude-User, and PerplexityBot. Google's AI Overviews and AI Mode draw from Google's normal index. One line in robots.txt or a stray noindex tag can hide your best page from all of them, and no amount of good writing gets past that. GPTBot and ClaudeBot are different: they collect training data, not answers, so blocking them is a reasonable choice that costs you nothing here. Be clear about the ceiling, too. Passing this doesn't get you cited, it only stops you being ruled out, and most sites we scan already pass. If this one is green, the answer is somewhere else on this list.

What you'll see

Technical Setup, in the Tracker

resilientniche.com/ai-visibility-tracker/site/bakingsubs.com
The Technical Setup page in the AI Visibility Tracker, showing four passed checks and a table of AI search crawlers that can all read the site
BakingSubs' Technical Setup, from its September 2026 scan. All four checks pass, so the signal reads Done, and every AI search crawler can read the site.

How it's scored

A checklist you can finish

Every site gets a 0 to 10 on this signal. It sits outside your content score as one of the two foundations, and when every check passes it scores 10 and it's done.

010

Blocked

0–4

A real access problem: robots.txt blocks a search crawler, or pages you want quoted carry noindex and can't be read at all.

Nearly

5–9

Reachable and indexable, but a check is still open: a page with no schema, no sitemap, or a minor canonical wrinkle.

Done

10

Every check passes. The signal is finished and scores 10; there is nothing left to do here.

How to improve it

The changes that move it

In the Tracker

Your Overview shows these four checks as a checklist, read straight off your site rather than judged by a model. Each open check says exactly what to change. When all four pass, the signal is marked done and scores 10, and a re-scan re-checks it.

  1. 1Remove any robots.txt rule that blocks OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User or PerplexityBot
  2. 2Take noindex off pages you want found; leave it on pages you hid on purpose
  3. 3Turn on the sitemap your site builder or SEO plugin already offers
  4. 4Add schema by template: Article for posts, Service or Product for offers, Organization for the business

If you're a local business

The access rules are the same for every site. What changes is the schema that matters: the type that names you as a real place an engine can recommend.

  • LocalBusiness schema, or its specific type (Dentist, Attorney, RoofingContractor and so on)
  • Your name, address, phone, opening hours and area served carried in that schema
  • Article or blog schema is never asked of you

Questions

Questions, answered

Which AI crawlers should I allow?

The search crawlers that fetch pages to answer questions: OAI-SearchBot and ChatGPT-User (ChatGPT), Claude-SearchBot and Claude-User (Claude), and PerplexityBot (Perplexity). Google's AI Overviews and AI Mode use normal Googlebot, so staying indexable covers those.

Should I block GPTBot and ClaudeBot?

That's your call, and it doesn't affect this score. GPTBot and ClaudeBot collect training data; they aren't the crawlers that fetch your page to answer a question. Blocking them costs you no citations.

Do I need schema markup to get cited by AI?

No. Structured data helps an engine understand what a page is, but it isn't required. A crawlable, well-written page can be cited with no schema at all. It's on the checklist because it is a one-time template change that removes ambiguity, not because it gets you quoted on its own.

I hid some pages with noindex on purpose. Will the scan nag me about them?

No. A page you've taken out of search is treated as out of play: we won't recommend content work on it or advise linking to it. The check only flags noindex so you can confirm it was deliberate.

Does blocking Google-Extended hurt my AI visibility?

Barely. Google-Extended only controls Gemini-app and Vertex grounding and training, not whether a page shows up in Google's AI search features. Those run off standard Googlebot indexing.

See where you stand on all six.

One scan scores your site on every signal, shows the pages an engine reads and skips, and hands you the fix worth making first.

3-day free trial · then $99/mo, cancel anytime