Free Robots.txt Generator with AI Crawler Controls
Build a clean robots.txt file for your WordPress site, and choose which AI crawlers you want to allow or block. Free, no signup, and it runs in your browser.
Only used to suggest your sitemap address.
Choosing a type fills in the rules below.
Rules for all crawlers
AI crawlers
Tick the ones you want to block. Anything you leave unticked follows the general rules above. Not sure what to pick? Use a preset.
Training bots
Search bots
User-triggered bots
Other bots
Sitemap
Upload this at the root of your domain so it opens at yourwebsite.com/robots.txt. If a robots.txt already exists, replace its contents and check the live file afterwards.
How to use the robots.txt generator
- Pick your site type. WordPress and WooCommerce get sensible starting rules. You can edit them.
- Decide what to do about AI crawlers. Leave them all unticked, block training bots only, or block everything. Each bot is explained in the list.
- Add your sitemap. Use the buttons to fill in the usual address, then check it opens on your site.
- Copy or download the file and upload it. It goes at the root of your domain. Then open it in a browser to check it.
What robots.txt does, and what it does not do
Robots.txt is a plain text file that tells crawlers which parts of your site they should not request. It is a polite request that well-behaved crawlers follow. It is not a security tool, and it does not remove pages from search results.
Blocking is not the same as hiding. A page blocked in robots.txt can still appear in Google if other sites link to it. To keep a page out of search, use a noindex tag instead and leave the page crawlable so Google can read the tag.
The AI crawlers in this tool
AI companies use different crawlers for different jobs. Some collect content for training, some build a search index so your pages can be cited, and some fetch a page when a person asks about it. Keeping these apart is the key to making a good decision.
| Bot | Company | Type | What it does |
|---|---|---|---|
| GPTBot | OpenAI | Training | Collects content that may be used for OpenAI model training. |
| OAI-SearchBot | OpenAI | Search | Indexes pages for ChatGPT search. |
| ChatGPT-User | OpenAI | User-triggered | Fetches a page when a ChatGPT user asks. |
| ClaudeBot | Anthropic | Training | Collects content that may be used for Claude training. |
| Claude-SearchBot | Anthropic | Search | Indexes pages for Claude search citations. |
| Claude-User | Anthropic | User-triggered | Fetches a page when a Claude user asks. |
| Google-Extended | Training control | Controls Gemini training use. It does not affect Search or AI Overviews. | |
| Applebot-Extended | Apple | Training control | Controls Apple Intelligence training use of content Applebot crawled. |
| PerplexityBot | Perplexity | Search | Builds the source index Perplexity cites. |
| Perplexity-User | Perplexity | User-triggered | Fetches pages live to answer a question. |
| meta-externalagent | Meta | Training | Used for Meta AI training and indexing. |
| CCBot | Common Crawl | Training | Feeds the open Common Crawl dataset. |
| Bytespider | ByteDance | Training | Poorly documented, and reported to ignore robots.txt. |
| Amazonbot | Amazon | Other | Used for Alexa and other Amazon services. |
Where these names come from: I could not open the official crawler pages of OpenAI, Anthropic and Google while building this tool. The names and descriptions come from two independent 2026 reference lists that agree with each other. Companies change their crawlers, so check each company's own documentation before you rely on a rule.
Should you block AI crawlers?
There is no single right answer, but it helps to separate three decisions.
- Search and user-triggered bots. If you want customers to find you through AI answers, I would not block these. Blocking them makes you harder to cite and harder to find.
- Training bots. This is a separate choice about whether your content can be used to train models. Blocking them does not change your Google rankings, so it is mostly a question of how you feel about reuse.
- Googlebot. Never block this one on a live site. It is what gets you into regular Google Search and AI Overviews.
A common middle path is to block the training bots and leave the search bots alone. The presets above do exactly that.
Where to add robots.txt in WordPress
- Rank Math. Go to Rank Math SEO, then General Settings, then Edit robots.txt, and paste the file.
- Yoast SEO. Go to Yoast SEO, then Tools, then File editor.
- Upload a file. Save the download as robots.txt and upload it to the root folder of your site through your hosting file manager.
WordPress also has a "Discourage search engines from indexing this site" setting under Settings, then Reading. That is a separate switch, so check it is off on a live site.
Mistakes to avoid
- Leaving "Disallow: /" on a live site. That tells every crawler to stay away from your whole site. It belongs on staging sites only.
- Blocking CSS and JavaScript folders. Google needs them to render your pages properly.
- Using robots.txt to hide private pages. The file is public, so anyone can read it. Use a password or a login instead.
- Blocking Googlebot when you meant Google-Extended. They do very different things.
- Forgetting the sitemap line. It helps crawlers find your pages.
Robots.txt is one small part of technical SEO. If you want the full picture, use my SEO audit checklist.
Frequently asked questions
Is this robots.txt generator free?
Yes. It is free, there is no signup, and you can generate as many files as you like.
Do you store my settings?
No. The file is built in your browser as you change the options. Nothing is sent to our server or saved, so when you close the tab it is gone. Copy or download your file before you leave.
Does blocking Google-Extended affect my Google rankings?
No. Google-Extended is a separate control for Gemini training. Googlebot still crawls your site, and Google Search and AI Overviews still use your content. Blocking Googlebot itself would remove you from regular search, so do not do that.
If I block GPTBot, will I disappear from ChatGPT?
Not necessarily. GPTBot is OpenAI's training crawler. ChatGPT search uses a different crawler called OAI-SearchBot. If you want to appear in ChatGPT answers, do not block OAI-SearchBot.
Do all AI bots obey robots.txt?
No. Robots.txt is a request, not a lock. Well-behaved crawlers follow it, but some bots have been reported to ignore it, and user-triggered fetchers may not follow it. For a hard block, use a firewall or a service such as Cloudflare.
Will robots.txt remove a page from Google?
No. A page blocked in robots.txt can still appear in search results if other sites link to it. To keep a page out of Google, use a noindex tag and leave the page crawlable so Google can see the tag.
Where does the robots.txt file go?
It must be at the root of your domain, at yourwebsite.com/robots.txt. In WordPress you can edit it from Rank Math under General Settings, from Yoast SEO under Tools, or by uploading a file to your site's root folder.
Why does this tool leave out crawl-delay?
Google ignores the crawl-delay rule, and most sites do not need it. If your server struggles with a particular bot, a firewall rule is a better fix.
How do I check my robots.txt after I upload it?
Open yourwebsite.com/robots.txt in a browser to see the live file. Google Search Console also has a robots.txt report that shows the version Google last fetched and any problems it found.
Not sure what your site should allow or block?
Robots.txt mistakes can quietly hide a site from Google. Our team can review your robots.txt, sitemap and indexing together and tell you what to fix first.