
Field Note: claude.ai/share/* and a documented robots.txt + noindex anti-pattern
Web CrawlingSEOConfiguration IssuesSecurity Best Practices
The post discusses an observation related to claude.ai/share/* URLs, noting their configuration in robots.txt and the use of X-Robots-Tag directives. It highlights a documented anti-pattern involving the combination of robots.txt and noindex instructions. The author suggests that the behavior being labeled as "working as designed" may have broader implications.