Skip to main content
aapp-hacks
← Back to articles
Author
app-hacks
Published
Updated
Reviewed

Cloudflare / robots.txt / SEO

Checking Cloudflare Content Signals Added to robots.txt After Deployment

A record of comparing the repository-managed robots.txt with the public response and confirming Cloudflare-managed content.

public/robots.txt is managed in the repository, but that does not guarantee the response at the public URL has identical content. When the response from https://app-hacks.net/robots.txt was checked on September 11, 2026, a block marked with a Cloudflare Managed content comment appeared before the site-managed Sitemap directive. This article records only the strings that were observed.

Separate repository-managed content from the public response

The repository file contains the following. The site is responsible for its Sitemap directive, which should match the URL generated by the Astro Sitemap integration.

User-agent: *
Allow: /

Sitemap: https://app-hacks.net/sitemap-index.xml

The public response, on the other hand, had a Cloudflare comment and additional rules for multiple user agents. It included this line:

Content-Signal: search=yes,ai-train=no,use=reference

This article does not infer the meaning of that line or its effect on crawlers. The important point is not to treat only the local public/robots.txt as the complete public content, and to check the final response delivered through the CDN separately.

How to check it and why

Retrieve the public URL with an HTTP client, then check its status, Content-Type, and body. Next, compare it with the repository’s public/robots.txt: confirm that the Sitemap line managed by the site remains and look for unintended changes. Do not infer from this check alone whether a setting exists to remove content added outside the repository.

robots.txt communicates the location of a Sitemap; it does not replace the Sitemap itself, a page canonical, or how a page appears in search. The public response checked here retained the Sitemap directive, so the next check is to retrieve /sitemap-index.xml and confirm that it includes only published articles.

Common pitfalls

Avoid treating the local file as authoritative without checking the public response. Conversely, do not commit CDN-added content as though it were a repository change. Results can change with configuration or service updates, so record the retrieval date and prioritize the actual response when checking again.

In this check, the block between the Cloudflare Managed content start and end comments also contained rules for multiple crawler names. Rather than reproducing every line and presenting it as a general policy, treat it as a fact about the public response on the date it was checked. The site-managed Allow and Sitemap lines stayed outside that block, making the distinction clear.

For a repeat check, retrieve the text over HTTP instead of only copying what a browser displays, then compare it with public/robots.txt. Looking at line breaks and comments helps avoid confusing a display difference with a rules difference. If changing robots.txt, first confirm that the Sitemap URL will not be broken before publishing.

Draft articles are excluded from the public Sitemap by the current implementation, so publishing an article also requires checking that its frontmatter publication setting and the Sitemap agree.