robots.txt states a content policy
One of 40 checks the scanner runs. This one is worth 1 point inside the AI Crawler Access dimension, which carries 16% of the total score.
What does this check look at?
robots.txt contains a Content-Signal directive with a value on the same line, for example `Content-Signal: search=yes, ai-input=yes, ai-train=no`. The directive is matched case-insensitively at the start of any line.
What does a failure mean?
No Content-Signal directive in robots.txt.
Weighted 1 of the 16 points in this dimension because adoption is early: the directive was proposed in 2025 and no major engine has committed to honouring it. It is scored because the convention is cheap to follow and a tool that recommends it should be able to show its own line, not because it moves a ranking.
How do I fix it?
- Add a line such as: Content-Signal: search=yes, ai-input=yes, ai-train=no. Early adoption and weighted low, but it states your position in the one file every crawler already reads.
Why is it worth 1 point?
Points are relative weights inside a dimension, and a dimension's weight decides how much of the total it can move. The reasoning for this dimension's weight is published rather than asserted:
| Dimension | AI Crawler Access — 16% of the total score |
| Points for this check | 1 of 23 in this dimension |
| Worth at most | 0.7 of the 100 points |
Highest weight of the twelve. A page that is blocked or unreadable has no path to being cited, so it dominates the total.
What else is measured in this dimension?
- robots.txt is served — 2 points
- No AI crawler is blocked — 6 points
- robots.txt declares a sitemap — 1 point
- Homepage returns HTTP 200 — 2 points
- Homepage is marked noindex — 2 points
- sitemap.xml is present and well-formed — 3 points
How do I see my own result?
Run a scan on your own domain. The report lists every failed check with the evidence that produced it, so you can tell a real failure from a check that does not apply.
Run a free GEO audit on your own site →
The full method, including the dimensions that are deliberately weighted low and the parts of the picture a single-URL scan cannot see, is on the methodology page.