How to block AI crawlers?
You can block AI crawlers by adding rules to your .htaccess file. The rules check the user agent, the name a crawler sends with each request, and return a 403 Forbidden error instead of your content. This article shows the rule, lists the user agents of common AI crawlers, and explains how to test the block and when robots.txt is enough.
How to block an AI crawler in .htaccess?
- Go to Site Tools > Site > File Manager.
- Click public_html in the folder tree on the left.
- Right-click the .htaccess file and choose Edit.
- Copy the current contents of the file to a text file on your computer as a backup.
- Paste the rule below at the very beginning of the file.
- Click Save in the toolbar.
Here is a sample rule to block a specific AI crawler by its user-agent:
This will prevent the ChatGPT-User bot from crawling your website. The [NC] flag makes the name match regardless of upper or lower case, and [F] returns a 403 Forbidden error (see the Apache mod_rewrite documentation). If you wish to block multiple providers, paste the same code and replace “ChatGPT-User” with another user-agent. You can also list several user agents in one rule, separated by a vertical bar (|):
If you wish to unblock that crawler in the future, simply remove the rules you added from your .htaccess file and save it.
The rules work on every SiteGround server, because .htaccess overrides are enabled on all of them. If NGINX Direct Delivery is on, static files such as images, CSS and JavaScript are served directly by NGINX and skip .htaccess, so the rules don’t apply to them. Pages served from Dynamic Cache, which is on by default, skip the rules too: once a page is in the cache, every visitor gets it, blocked crawlers included. After the cache is flushed, the rules apply again until a visitor who isn’t blocked loads the page and it’s cached again. A robots.txt rule doesn’t have this limit, because a crawler that follows robots.txt reads it before it requests a page.
Common AI crawler user agents
Most AI companies run separate crawlers for model training, for their search features and for pages a user asks about, so you can block one purpose and keep another. The names below come from each company’s documentation: OpenAI, Anthropic, Perplexity, Common Crawl and Google. Check them for changes before you add a rule.
- GPTBot (OpenAI): collects content that may be used to train OpenAI’s AI models.
- OAI-SearchBot (OpenAI): finds websites for ChatGPT search. Sites that block it aren’t shown in ChatGPT search answers.
- ChatGPT-User (OpenAI): visits a page when a ChatGPT user asks about it. It may not follow robots.txt.
- ClaudeBot (Anthropic): collects content that could be used to train Anthropic’s AI models.
- Claude-SearchBot (Anthropic): indexes content to improve Claude’s search results.
- Claude-User (Anthropic): fetches a page when a Claude user asks a question.
- PerplexityBot (Perplexity): finds and links websites in Perplexity search results. Perplexity says it isn’t used to train AI models.
- Perplexity-User (Perplexity): visits a page to answer a Perplexity user’s question. It generally ignores robots.txt.
- CCBot (Common Crawl): collects pages for Common Crawl’s open archive of web data.
- Google-Extended (Google): not a separate crawler, but a robots.txt token. It controls whether content Google crawls can be used to train and ground Gemini models. It has no user agent of its own, so an .htaccess rule can’t block it; use robots.txt instead.
Blocking a search or user crawler (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User) can keep your site out of that AI assistant’s answers.
How to block AI crawlers with robots.txt?
Crawlers that follow the Robots Exclusion Protocol can be stopped with a robots.txt file in your public_html/ folder, without any .htaccess rules. For example, to stop GPTBot and Google-Extended, add:
robots.txt is a request, not a block. Crawlers that ignore it, such as ChatGPT-User and Perplexity-User, can still reach your site, so use an .htaccess rule for them. For Google-Extended, robots.txt is the only option.
How to check that the block works?
Request your site with the crawler’s user agent from a terminal on your computer. Replace yourdomain.com with your domain:
If the block works, the first line of the response shows 403. Run the command again without -A "ChatGPT-User" to make sure regular visitors still get your site: the response should show 200 or a redirect, such as 301.
Frequently asked questions
Will blocking AI crawlers affect my Google Search rankings?
The rules only affect the user agents you list, so don’t add Googlebot, which Google Search uses. Blocking Google-Extended in robots.txt doesn’t affect your site’s inclusion or ranking in Google Search.
Why does the test still return 200 for a blocked crawler?
Check that the rules are at the very beginning of the .htaccess file in the public_html/ folder of that website, that the file includes the RewriteEngine On line (it turns on mod_rewrite, which the rules need), and that the user agent is spelled as in the crawler’s documentation. If the response includes x-proxy-cache: HIT, the page came from Dynamic Cache, which skips the rules. Flush the cache and test again: the blocked user agent gets 403 until a visitor who isn’t blocked loads the page and it’s cached again.
What if my website has no .htaccess file?
Create one. In File Manager, click public_html, click New File in the top toolbar, enter .htaccess as the name and click Confirm. Then add the rules.
My website shows an error after I edited .htaccess. What should I do?
Restore the backup copy of the file, then add the rule again and check it for typos. If regular visitors get a 403 Forbidden error, see How to fix the 403 Forbidden error.