SEO · robots.txt · AI crawlers

robots.txt and AI crawler check

Check whether a domain has robots.txt and llms.txt, whether a sitemap is listed and how common search and AI crawlers are allowed to access the site root.

robots.txt guides crawler behavior. llms.txt can provide a machine-readable summary of the site and its important resources for AI systems.

Enter a domain, for example nettools.fi.
Guide · robots.txt and AI crawlers

What does this tool check?

The tool fetches /robots.txt and /llms.txt for the domain. It checks sitemap lines and evaluates how common crawlers such as Googlebot, Bingbot, GPTBot, ChatGPT-User and OAI-SearchBot may access the site root.

robots.txt

robots.txt tells crawlers which paths they may or may not fetch. It is not access control, but it is an important signal for search engines.

AI crawlers

GPTBot, ChatGPT-User, OAI-SearchBot and other AI crawlers can be allowed or blocked with robots.txt rules.

llms.txt

llms.txt is a text file placed at the site root to describe important content and resources for AI systems.

Sitemap

robots.txt can include a sitemap line to help search engines discover the indexable URLs of the site.

How to interpret the result

If robots.txt is missing, it usually does not mean crawling is blocked. It means no separate robots.txt restrictions were found. If a crawler hasDisallow: /, that crawler is blocked from the site root.

llms.txt is optional, but it can help describe the purpose of a site, its main tools and important URLs in a machine-readable way.