robots.txt is served
One of 40 checks the scanner runs. This one is worth 2 points inside the AI Crawler Access dimension, which carries 16% of the total score.
What does this check look at?
A request for /robots.txt returns HTTP 200 with a non-empty body. A 200 with an empty or whitespace-only body counts as not served, because it publishes no policy at all.
What does a failure mean?
No readable robots.txt at the domain root.
How do I fix it?
- Publish robots.txt. With no robots.txt every crawler is permitted by default, so this is a policy gap rather than a block.
Why is it worth 2 points?
Points are relative weights inside a dimension, and a dimension's weight decides how much of the total it can move. The reasoning for this dimension's weight is published rather than asserted:
| Dimension | AI Crawler Access — 16% of the total score |
| Points for this check | 2 of 23 in this dimension |
| Worth at most | 1.4 of the 100 points |
Highest weight of the twelve. A page that is blocked or unreadable has no path to being cited, so it dominates the total.
What else is measured in this dimension?
- No AI crawler is blocked — 6 points
- robots.txt declares a sitemap — 1 point
- Homepage returns HTTP 200 — 2 points
- Homepage is marked noindex — 2 points
- sitemap.xml is present and well-formed — 3 points
- robots.txt states a content policy — 1 point
How do I see my own result?
Run a scan on your own domain. The report lists every failed check with the evidence that produced it, so you can tell a real failure from a check that does not apply.
Run a free GEO audit on your own site →
The full method, including the dimensions that are deliberately weighted low and the parts of the picture a single-URL scan cannot see, is on the methodology page.