How do you check if AI can read your website?
Short answer
Open yourwebsite.com/robots.txt and look for AI crawler names such as GPTBot, OAI-SearchBot, PerplexityBot, Google-Extended or ClaudeBot followed by "Disallow: /". If you see that, the crawler is blocked. Then paste your address into Google's Rich Results Test to see whether your pages have schema markup.
Check 1: are AI crawlers blocked in your robots.txt?
- Type your domain followed by
/robots.txt, for examplehttps://www.example.ae/robots.txt. The file always lives at the root of the domain, not inside a folder like/en/. - Look for lines that start with
User-agent:. That is the name of the crawler the rule applies to. - A rule of
Disallow: /under a crawler's name means it cannot read anything.Allow: /or an emptyDisallow:means it can. - Check the names that matter for AI:
GPTBot,OAI-SearchBotandChatGPT-User(ChatGPT),PerplexityBot(Perplexity),Google-Extended(Google's AI models),ClaudeBot(Claude) andCCBot(Common Crawl). - Check
User-agent: *too. The star means every crawler, soDisallow: /there blocks the whole site.
If the file does not exist, crawlers are usually allowed by default. Blocks are often added by accident by a developer or a security plugin, so it is worth checking even if you never touched the file.
Check 2: does your website have schema markup?
- Open Google's Rich Results Test.
- Paste your page address and run the test.
- Items under "Detected structured data", such as Local business, FAQ or Breadcrumbs, mean the page has schema. "No items detected" means it has none Google can use.
- For a full list of every label on the page, also try the Schema Markup Validator.
- Repeat for your key inner pages, not only the homepage.
Which schema types should a UAE business have?
- LocalBusiness, or a specific type such as Restaurant, Dentist or RealEstateAgent, with your name, address, phone and opening hours.
- FAQPage for an FAQ section that is also visible on the page.
- Organization with your logo and links to your official profiles, such as LinkedIn and Instagram.
- Product or Service where you list what you sell.
What else stops AI from reading a page?
Many AI crawlers read only the basic HTML of a page. Content that appears only after heavy JavaScript runs, or that sits behind pop-ups and logins, may be missed. Keep your key facts, like services, prices, location and hours, as plain text on the page.