Add Disallow rules for the most aggressive AI training crawlers to your robots.txt with a single toggle in the admin.
How it works
ClassifAI extends the WordPress robots_txt filter so that, when the feature is enabled, the rendered robots.txt at /robots.txt includes Disallow rules for the most common AI training and scraping bots — including GPTBot, Google-Extended, ClaudeBot, CCBot, Bytespider, anthropic-ai, and the named user agents from other large-model providers. The list is curated and updated as new bots are identified; administrators do not have to maintain it themselves.
Because the rules are written through the standard WordPress robots.txt filter pipeline, they cooperate with whatever else is already in your robots.txt — your existing search-engine rules, the WordPress core rules, and any rules added by SEO plugins all stay in place.
Configuration
This is a single-toggle feature: enable it and the rules appear; disable it and they disappear. There is no provider involvement and no outbound API call — the entire effect is a server-side rule on a static text file.
What it doesn’t do
This feature is one of the few in ClassifAI that has nothing to do with adding AI to your site and everything to do with limiting what other AIs can do with your site. It relies on bot operators honouring robots.txt — a convention, not an enforcement mechanism. For stronger guarantees, pair it with a CDN or WAF rule that blocks the same user agents at the edge.