WordPress robots.txt: where it comes from, what it blocks and how to fix it
WordPress robots.txt is a short text at yourdomain.com/robots.txt that tells Google's robots what not to visit. Usually WordPress creates it on the fly. When it blocks the site, the cause is most often a real file left on the server after a move, or a rule added in an SEO plugin. While you are at it, check the visibility option in Settings > Reading.
Author: The TellMyShop team. A guide for WordPress site and WooCommerce store owners: the cause, the manual fix and the same work in a chat conversation.
Last updated: · 7 min read
How to check in a minute whether robots.txt blocks your site
Open your site's address with /robots.txt added in a browser and read what is there. The file has a few lines and is easier to read than it looks.
| What you see in the file | What it means | Is it a problem |
|---|---|---|
Disallow: /wp-admin/ and Allow: /wp-admin/admin-ajax.php | the default WordPress file, the admin is closed to robots | no, that is how it should be |
Disallow: / under User-agent: * | a ban on visiting the whole site | yes, the whole site is blocked |
Disallow: /shop/ or Disallow: /product/ | a ban for one section | yes, if that section should be in Google |
Disallow: /*? | a ban for addresses with parameters | it depends, sometimes deliberate for store filters |
a Sitemap: line with the sitemap address | a pointer to where the sitemap is | no, it helps |
| an error page or a blank page | there is no file | no, Google assumes it may visit everything |
Google explains what this file is for and what it can't do in its series on robots.txt on the Search Central blog. The key point from there: robots.txt controls a robot's visits, not whether a page ends up in results.
If the file looks fine and the site still isn't in Google, the cause lies elsewhere. You will find the full list in the post on why a WordPress site is not showing in Google.
Where does WordPress get robots.txt from?
From three places, and that is the source of most of the confusion. You edit one, and Google reads another. Next to them is a fourth place people confuse with robots.txt: the “Discourage search engines from indexing this site” box in Settings > Reading. In current WordPress versions it does not add a block to robots.txt, it adds a noindex tag to pages instead, and the effect is similar: the site drops out of Google.
- The on-the-fly version from WordPress. When there is no real file on the server, WordPress builds robots.txt on every visit. The content depends on the settings, and plugins can add rules to it.
- Rules from plugins. An SEO plugin or a store plugin can add its own lines to the on-the-fly version, and some SEO plugins keep the whole robots.txt content in their settings.
- A real file in the main folder. A developer, the host or an SEO plugin uploaded it. When it exists, the server serves it instead of the WordPress version, and everything you set in the admin stops mattering.
SEO plugins add their own editors. Yoast SEO has a file editor in its tools and creates a real file on the server, as described in its help on the robots.txt file in Yoast SEO. Rank Math works the other way round: it keeps the rules in its settings and serves them on the fly, but only when there is no real file on the server. It describes this in its article on editing robots.txt in Rank Math. When you switch from one plugin to the other, the old file often stays and keeps winning.
How to fix robots.txt in WordPress step by step
Start with the place that is most often to blame, and check the next ones only when the previous one is fine.
- Settings > Reading. Make sure the box about discouraging search engines from indexing is unticked. The box doesn't change robots.txt, but it is the most common reason the whole site disappears from Google, so check it first.
- Find the real file. In your host's file manager or over FTP, look in the site's main folder, where the wp-content folder is. If robots.txt is there, that is what goes to Google, not the WordPress version. If there is no file, a plugin is adding the block, so look for it in your SEO plugin's robots.txt editor.
- Make a backup. Download the current file to your computer before you change anything. If something goes wrong, you upload it back.
- Fix or delete the file. Remove the blocking line, or the whole file if you prefer the WordPress and SEO plugin version. In Yoast SEO you can change the same file from the plugin's editor.
- Clear the cache. A cache plugin, the host or a CDN can keep serving the old version for a while. Open the address in a private window.
- Tell Google. In Search Console there is a robots.txt report in the settings. Request a new fetch of the file there, then check the home page and one product page.
Watch out for the reflex “I'll block it so it disappears from Google”. A block in robots.txt means the robot won't see the noindex on the page, so the address may stay in results. How to untangle this is covered in the post on a page excluded by the noindex tag.
How the chat checks robots.txt and fixes it with a backup of the file
The TellMyShop plugin for WordPress and WooCommerce connects your site to the chat. In the free TellMyShop Free the chat checks a given page the way Google sees it: the response code, noindex in the SEO plugin and in the Reading option, the page headers, robots.txt rules, the sitemap. At the top of the answer it gives the most likely cause. A review of the whole site also checks whether the site is asking search engines not to index it.
In TellMyShop Pro the chat also reads the site's files, so it sees whether there is a real robots.txt file on the server and what is in it. It makes the fix where the cause is, and whether it can do so at all is decided by the switches in the plugin panel:
| Where the cause is | Who can fix it | What TellMyShop Pro needs |
|---|---|---|
| the option in Settings > Reading | you in the admin or the chat | the Theme switch, which covers site settings (24 hours, needs a database backup from today) |
| a real file in the main folder | you over FTP or the chat | the Service switch, full technical access (3 hours), a backup of the file before the change |
| rules in the SEO plugin settings | you in the plugin's editor or the chat | the Plugins switch, which covers plugin settings (24 hours) |
You: Search Console says robots.txt is blocking the store. Check where it's coming from.
Your chat: I checked the home page and one product page. The option in Settings > Reading is fine, there is no noindex. The block comes from a real robots.txt file in the site's main folder, dated before the move from the test version. I suggest this change:
Before saving I'll make a backup of the current file. Saving needs the Service switch. Save? (yes / no)
You: I've turned on Service. Yes.
Your chat: Saved. I checked /robots.txt live, the new content is visible. Now request a new fetch of the file in Search Console.
Card status: waiting for your “yes”old version: User-agent: *
old version: Disallow: /
new version: User-agent: *
new version: Disallow: /wp-admin/
new version: Allow: /wp-admin/admin-ajax.php
new version: Sitemap: https://yourstore.com/sitemap_index.xml
The sitemap address in the example is illustrative, the chat enters the one your SEO plugin gives. Every change goes into the change history, and the file can be restored from the backup. You can undo most changes with one instruction in the conversation.
What TellMyShop doesn't do: it doesn't click in Search Console for you and it can't speed up how quickly Google refreshes its copy of the file. It also doesn't change the .htaccess file, which is sometimes a second place for blocks. That stays with your host or developer.
Start with the free TellMyShop Free, which will find the block. Works with Claude or ChatGPT, including free accounts. Details about the plugin are on the TellMyShop for WordPress and WooCommerce page, and Pro prices on the pricing page. If you want a wider checklist, read the post on a WordPress SEO audit.
Questions and answers
Where is the robots.txt file in WordPress?
Most often nowhere, because WordPress creates it on the fly when someone opens yourdomain.com/robots.txt. A real file sits in the site's main folder only when someone uploaded it there or an SEO plugin created it. If such a file exists, it wins over the WordPress version.
Does robots.txt remove a page from Google?
No. Robots.txt tells robots what not to visit, but a blocked address can still appear in results, just without a description. For a page to disappear from Google you need noindex, and the robot must be able to read it, so you don't block the address in robots.txt.
What does Disallow: / mean in robots.txt?
It forbids robots from visiting the whole site. On a live site it is almost always a mistake. In WordPress it usually appears after the site was moved from a test version along with its file, or after a rule was added in an SEO plugin.
After the fix Google still shows the block. What should I do?
Google keeps a copy of robots.txt for a while. In Search Console, open the robots.txt report in the settings and request a new fetch of the file, then check the most important addresses with the URL inspection tool. Also check that your host or CDN isn't serving an old cached version.
Will the TellMyShop plugin change robots.txt by itself?
In the free TellMyShop Free the chat checks whether robots.txt rules block the site and tells you where they come from, but changes nothing. In TellMyShop Pro it fixes the cause after you agree: the visibility option in the Reading settings with the Theme switch, or the real file on the server with the Service switch, always with a backup of the file before the change.
WordPress is a trademark of the WordPress Foundation. WooCommerce is a trademark of Automattic Inc. Claude is a trademark of Anthropic PBC. ChatGPT and OpenAI are trademarks of OpenAI. TellMyShop is not affiliated with these companies.