We make an ordinary public request.
The check validates the address, resolves only public network targets, follows up to three safe redirects and reads at most 128 KB of robots.txt. It does not impersonate a provider crawler.
Free AI crawler access checker
See the policy your robots file declares.
Check one public page or site. We read its public robots.txt file, apply the published matching rules and show the exact evidence.
One public address. Declared policy only. No email required.
An allowed robots rule does not prove a crawler can reach, index, retrieve, cite, recommend or use a page.
A clear reading of a public robots file.
Declared crawler policy
Enter a public site or page address.
We use its path when we interpret the site's public robots.txt file. Use a page anyone can open without signing in.
The method
A cautious read of a public file.
We show the matching rule.
The parser applies named user-agent groups before wildcard groups. The longest matching path wins, with Allow winning a tie. You can open the source line and the raw file.
You keep the boundary in view.
Provider documentation describes roles, while the result only describes the declared robots policy. Access and visibility still need their own evidence.
FAQ
Before you use the result
Does Allowed mean an AI product will show my page?
No. Allowed means this parser found an Allow rule for a documented token and path. It does not prove that a provider reached the site, indexed it, retrieved it, cited it, recommended it or used it for model development.
Why does a user-triggered row sometimes say robots may not apply?
Provider documentation differs. For example, OpenAI and Perplexity describe user-requested fetches that can operate differently from automated crawling. The row still shows the declared file policy, then points you to the provider's documentation for the operating boundary.
What does Unspecified mean?
A matching user-agent group had no Allow or Disallow rule for this path, or the file had no matching named or wildcard group. It is not an access decision and it is not visibility evidence.
Why could the robots file be unreachable?
The public request may time out, reject the request, exceed the safety limit or fail a safe redirect or DNS check. That result does not tell you how an AI provider would respond.
Do you save the address or result?
We use the address and public response for this request, without putting either in analytics or run records. The rate-limit receipt holds no address, response body, redirect or response header. Your result stays in this tab unless you save it.