robots.txt is a small text file in your site's root directory that tells search engine bots which addresses they may enter and which they should not. A single line mistake can stop the crawling of your entire site; a correctly written file, on the other hand, directs crawl resources toward your important pages. In this guide we explain what the robots.txt file does, how to write it, which directives Google supports, and the most common mistakes, with examples.
What is robots.txt?
robots.txt is the file of the standard known as the "Robots Exclusion Protocol." It is a plain text file (UTF-8) and must
be located at the root of the domain: https://www.example.com/robots.txt.
A robots.txt placed in a subfolder is not taken into account. Each subdomain (for example blog.example.com)
and each protocol/port combination needs its own file.
The purpose of the file is to manage crawl traffic: preventing bots from spending time on areas of no value, such as admin panels, carts, internal search results, or endless filtered URL variations. It is not a security tool; anyone can read the file, so writing the name of a folder you want to keep secret here only exposes it. Use a password or authorization for hidden content.
How do search engines use robots.txt?
Before crawling any page on a site, a bot requests the robots.txt file and reads the rules inside it. Rules are grouped by "user-agent"; the bot selects the group that suits it best and applies only that one. A few important details:
- If the file cannot be found (a 4xx response such as 404), Google interprets this as "no restrictions" and crawls the site freely.
- If the server returns a 5xx error, Google slows or stops crawling cautiously; so it is important that the robots.txt address responds healthily.
- Google processes only the first 500 KiB of the file; very large files are truncated.
- Path rules are case-sensitive:
/Admin/and/admin/are different paths. - Google does not re-download the file on every request; it keeps it in cache. The effect of a change you make appears after a while, not immediately.
Basic syntax and directives
The directives Google supports are limited to four. The table below summarizes what each one does.
| Directive | What does it do? | Example |
|---|---|---|
User-agent | Specifies which bot the rule belongs to; * means all bots. | User-agent: Googlebot |
Disallow | Prevents crawling of the specified path. | Disallow: /cart/ |
Allow | Permits a specific path inside a blocked folder. | Allow: /cart/help.html |
Sitemap | Declares the full address of the sitemap file; it is independent of groups. | Sitemap: https://www.example.com/sitemap.xml |
Two wildcard characters can be used: * (any sequence of characters) and $ (the end of the address).
For example, Disallow: /*.pdf$ covers every path that ends with .pdf, and Disallow: /*?sort=
covers addresses with the "sort" parameter. If an address matches both an Allow and a Disallow rule, Google applies the
more specific (longer) rule.
Unsupported directives: Crawl-delay is ignored by Google (some other search engines may read it). The Noindex
line has also not been valid inside robots.txt since 2019. To remove a page from the index, you need to add a noindex meta tag or HTTP header to the page.
An example robots.txt file
For a corporate site or a small e-commerce site, a reasonable starting template could look like this:
User-agent: *
Disallow: /admin/
Disallow: /cart/
Disallow: /checkout/
Disallow: /search/
Disallow: /*?sort=
Disallow: /*?session=
Sitemap: https://www.example.com/sitemap.xml
This file tells all bots not to crawl the admin, cart, checkout, and internal search pages, as well as the addresses with sorting and session
parameters; it allows everything else. A rule commonly seen on WordPress sites is the pair Disallow: /wp-admin/ and
Allow: /wp-admin/admin-ajax.php. To determine which paths are worthless on your own site, first look at the crawl statistics
in Search Console and your server logs; ready-made templates copied without looking often block more than necessary.
Separate rules for different bots
You can define multiple groups in the file. Each group starts with one or more User-agent lines, followed by the rules. The bot
selects the single group that best matches its name; it does not additionally apply the rules of the general (*) group. That is why,
when you write a group for a specific bot, you need to repeat the shared rules in that group as well.
User-agent: *
Disallow: /admin/
Disallow: /search/
User-agent: Googlebot-Image
Disallow: /private-images/
Disallow: /admin/
Sitemap: https://www.example.com/sitemap.xml
In the example above, Googlebot-Image applies only its own group; that is why the /admin/ rule also appears in that group. Putting
blank lines between groups improves readability. You can write comments by starting a line with #; these lines are ignored. Do not
overcrowd the rules: a neglected robots.txt that has swollen over the years becomes a risk source where nobody knows why a given rule is there. Writing
a short comment next to each rule explaining its reason is a good habit.
Blocking crawling is not the same as removing from the index
This is the most frequently confused point. robots.txt only restricts crawling. If a blocked address receives links from other sites or internal links, Google can still add the address to the index without ever seeing the content; in results it appears with the address alone and no description. Worse, if you both block a page with robots.txt and place a noindex on it, Google cannot see the noindex because it cannot access the page.
| Your goal | Correct method |
|---|---|
| Do not spend crawl budget on worthless addresses | Disallow in robots.txt |
| Keep the page from appearing in search results | Leave the page open to crawling and add a noindex tag |
| Keep the content hidden from everyone | Password/authorization (robots.txt does not provide security) |
| Consolidate the signals of duplicate addresses | 301 redirect or canonical (see our canonical guide) |
Small site or large site? On a site with a few hundred pages, robots.txt has a limited effect on crawl efficiency; its real value lies in keeping the wrong areas out and announcing the sitemap. For large e-commerce or classified sites with hundreds of thousands of addresses, however, removing unnecessary address combinations from crawling really makes a difference.
The most common mistakes
- Blocking the entire site. If a
Disallow: /line arrives with files carried over from the development environment to the live site, the site becomes uncrawlable. Always check robots.txt when publishing a new version. - Blocking CSS and JavaScript folders. Google needs these files to render the page and evaluate mobile usability. If they are blocked, the page may appear incomplete.
- Blocking addresses that are in the sitemap. Saying "add this to the index" on one hand and "do not enter" on the other is inconsistent; Search Console shows this as a warning.
- Writing noindex in robots.txt. As explained above, it no longer works.
- Placing it in the wrong location. If the file is not at the root, or its name is different, such as
Robots.TXT, the rules are not counted. - Writing a path that is too broad.
Disallow: /productcovers not only the "product" folder but every address starting with "/product", such as/products/and/product-details.html. If you mean a folder, put a trailing slash at the end. - Blocking pages from crawling and then expecting a 301. If you block old addresses that are being redirected, the bot cannot see the redirect.
Testing before going live
Before changing the file, take a backup of the current version. Then follow these steps:
- Open the file in your browser (
/robots.txt) and confirm that it loads as plain text with a 200 status code. - Open the Settings → robots.txt report in Search Console; it shows the version Google last downloaded, along with syntax errors and warnings.
- Use the URL Inspection tool for several important pages; if you see the message "Blocked by robots.txt," review the rule.
- In the week after publishing, check the Pages report for an unexpected increase in the count under "Blocked by robots.txt." Our Google indexing problems guide will help you diagnose general indexing issues.
For official documentation, see the robots.txt guide on Google Search Central. robots.txt is only one item in the overall health check; for all items, use our technical SEO checklist.
Frequently asked questions
Is robots.txt mandatory?
No. If the file does not exist, bots crawl the site freely. However, if there are areas you want to block or you want to announce the sitemap address, creating one is useful.
When will a change to robots.txt take effect?
Google keeps the file in cache and usually refreshes it within a day at most; so the effect may become visible anywhere from a few hours to a day. In urgent cases, a recrawl can be requested from the Search Console robots.txt report.
Can I block AI bots with robots.txt?
Yes, for bots that follow the rules; each bot has its own user-agent name and can be written as a separate group. However, robots.txt is a request, not an enforcement mechanism; it will not stop bots that ignore the rules.
Why is a page I blocked with Disallow still appearing on Google?
Because Disallow is not a command to remove from the index. The address may be listed without content because it receives links from elsewhere. Solution: remove the block and add a noindex to the page, or remove the page entirely.
Conclusion
robots.txt is a small file with big consequences. The rule is simple: block only the areas that are genuinely worthless, do not touch CSS/JS files, add the sitemap address, and do not use it as a way to remove pages from the index. Verifying the file in Search Console after each release and making sure your important pages can be crawled prevents most disasters from the start. As your next step, you can move on to our XML sitemap guide to prepare your sitemap file.