# robots.txt states a content policy

How the robots.txt states a content policy check works, what a failure means, and how to fix it in the AI Crawler Access dimension.

- Dimension: AI Crawler Access (16% of the total score)
- This check: 1 point
- Check id: `content-signal`

## What does this check look at?

robots.txt contains a Content-Signal directive with a value on the same line, for example `Content-Signal: search=yes, ai-input=yes, ai-train=no`. The directive is matched case-insensitively at the start of any line.

## What does a failure mean?

No Content-Signal directive in robots.txt.

> Weighted 1 of the 16 points in this dimension because adoption is early: the directive was proposed in 2025 and no major engine has committed to honouring it. It is scored because the convention is cheap to follow and a tool that recommends it should be able to show its own line, not because it moves a ranking.

## How do I fix it?

- Add a line such as: Content-Signal: search=yes, ai-input=yes, ai-train=no. Early adoption and weighted low, but it states your position in the one file every crawler already reads.

## Why is it worth 1 points?

Highest weight of the twelve. A page that is blocked or unreadable has no path to being cited, so it dominates the total.

## What else is measured in this dimension?

- [robots.txt is served](https://geo-scanner.ccie13192.com/checks/robots-present/) — 2 points
- [No AI crawler is blocked](https://geo-scanner.ccie13192.com/checks/robots-ai-allowed/) — 6 points
- [robots.txt declares a sitemap](https://geo-scanner.ccie13192.com/checks/robots-sitemap/) — 1 point
- [Homepage returns HTTP 200](https://geo-scanner.ccie13192.com/checks/home-200/) — 2 points
- [Homepage is marked noindex](https://geo-scanner.ccie13192.com/checks/home-noindex/) — 2 points
- [sitemap.xml is present and well-formed](https://geo-scanner.ccie13192.com/checks/sitemap-valid/) — 3 points

## How do I see my own result?

Run a free scan on your own domain at https://geo-scanner.ccie13192.com/ - the report lists every failed check with the evidence that produced it.

The full method is published at https://geo-scanner.ccie13192.com/methodology/