Skip to content
EN · $
Panel Login Buy Package
Technical SEO

What Is robots.txt? How to Write It, With Examples and Common Mistakes

6 min read

robots.txt is a small text file in your site's root directory that tells search engine bots which addresses they may enter and which they should not. A single line mistake can stop the crawling of your entire site; a correctly written file, on the other hand, directs crawl resources toward your important pages. In this guide we explain what the robots.txt file does, how to write it, which directives Google supports, and the most common mistakes, with examples.

What is robots.txt?

robots.txt is the file of the standard known as the "Robots Exclusion Protocol." It is a plain text file (UTF-8) and must be located at the root of the domain: https://www.example.com/robots.txt. A robots.txt placed in a subfolder is not taken into account. Each subdomain (for example blog.example.com) and each protocol/port combination needs its own file.

The purpose of the file is to manage crawl traffic: preventing bots from spending time on areas of no value, such as admin panels, carts, internal search results, or endless filtered URL variations. It is not a security tool; anyone can read the file, so writing the name of a folder you want to keep secret here only exposes it. Use a password or authorization for hidden content.

How do search engines use robots.txt?

Before crawling any page on a site, a bot requests the robots.txt file and reads the rules inside it. Rules are grouped by "user-agent"; the bot selects the group that suits it best and applies only that one. A few important details:

  • If the file cannot be found (a 4xx response such as 404), Google interprets this as "no restrictions" and crawls the site freely.
  • If the server returns a 5xx error, Google slows or stops crawling cautiously; so it is important that the robots.txt address responds healthily.
  • Google processes only the first 500 KiB of the file; very large files are truncated.
  • Path rules are case-sensitive: /Admin/ and /admin/ are different paths.
  • Google does not re-download the file on every request; it keeps it in cache. The effect of a change you make appears after a while, not immediately.

Basic syntax and directives

The directives Google supports are limited to four. The table below summarizes what each one does.

DirectiveWhat does it do?Example
User-agentSpecifies which bot the rule belongs to; * means all bots.User-agent: Googlebot
DisallowPrevents crawling of the specified path.Disallow: /cart/
AllowPermits a specific path inside a blocked folder.Allow: /cart/help.html
SitemapDeclares the full address of the sitemap file; it is independent of groups.Sitemap: https://www.example.com/sitemap.xml

Two wildcard characters can be used: * (any sequence of characters) and $ (the end of the address). For example, Disallow: /*.pdf$ covers every path that ends with .pdf, and Disallow: /*?sort= covers addresses with the "sort" parameter. If an address matches both an Allow and a Disallow rule, Google applies the more specific (longer) rule.

Unsupported directives: Crawl-delay is ignored by Google (some other search engines may read it). The Noindex line has also not been valid inside robots.txt since 2019. To remove a page from the index, you need to add a noindex meta tag or HTTP header to the page.

An example robots.txt file

For a corporate site or a small e-commerce site, a reasonable starting template could look like this:

User-agent: *
Disallow: /admin/
Disallow: /cart/
Disallow: /checkout/
Disallow: /search/
Disallow: /*?sort=
Disallow: /*?session=

Sitemap: https://www.example.com/sitemap.xml

This file tells all bots not to crawl the admin, cart, checkout, and internal search pages, as well as the addresses with sorting and session parameters; it allows everything else. A rule commonly seen on WordPress sites is the pair Disallow: /wp-admin/ and Allow: /wp-admin/admin-ajax.php. To determine which paths are worthless on your own site, first look at the crawl statistics in Search Console and your server logs; ready-made templates copied without looking often block more than necessary.

Separate rules for different bots

You can define multiple groups in the file. Each group starts with one or more User-agent lines, followed by the rules. The bot selects the single group that best matches its name; it does not additionally apply the rules of the general (*) group. That is why, when you write a group for a specific bot, you need to repeat the shared rules in that group as well.

User-agent: *
Disallow: /admin/
Disallow: /search/

User-agent: Googlebot-Image
Disallow: /private-images/
Disallow: /admin/

Sitemap: https://www.example.com/sitemap.xml

In the example above, Googlebot-Image applies only its own group; that is why the /admin/ rule also appears in that group. Putting blank lines between groups improves readability. You can write comments by starting a line with #; these lines are ignored. Do not overcrowd the rules: a neglected robots.txt that has swollen over the years becomes a risk source where nobody knows why a given rule is there. Writing a short comment next to each rule explaining its reason is a good habit.

Blocking crawling is not the same as removing from the index

This is the most frequently confused point. robots.txt only restricts crawling. If a blocked address receives links from other sites or internal links, Google can still add the address to the index without ever seeing the content; in results it appears with the address alone and no description. Worse, if you both block a page with robots.txt and place a noindex on it, Google cannot see the noindex because it cannot access the page.

Your goalCorrect method
Do not spend crawl budget on worthless addressesDisallow in robots.txt
Keep the page from appearing in search resultsLeave the page open to crawling and add a noindex tag
Keep the content hidden from everyonePassword/authorization (robots.txt does not provide security)
Consolidate the signals of duplicate addresses301 redirect or canonical (see our canonical guide)

Small site or large site? On a site with a few hundred pages, robots.txt has a limited effect on crawl efficiency; its real value lies in keeping the wrong areas out and announcing the sitemap. For large e-commerce or classified sites with hundreds of thousands of addresses, however, removing unnecessary address combinations from crawling really makes a difference.

The most common mistakes

  1. Blocking the entire site. If a Disallow: / line arrives with files carried over from the development environment to the live site, the site becomes uncrawlable. Always check robots.txt when publishing a new version.
  2. Blocking CSS and JavaScript folders. Google needs these files to render the page and evaluate mobile usability. If they are blocked, the page may appear incomplete.
  3. Blocking addresses that are in the sitemap. Saying "add this to the index" on one hand and "do not enter" on the other is inconsistent; Search Console shows this as a warning.
  4. Writing noindex in robots.txt. As explained above, it no longer works.
  5. Placing it in the wrong location. If the file is not at the root, or its name is different, such as Robots.TXT, the rules are not counted.
  6. Writing a path that is too broad. Disallow: /product covers not only the "product" folder but every address starting with "/product", such as /products/ and /product-details.html. If you mean a folder, put a trailing slash at the end.
  7. Blocking pages from crawling and then expecting a 301. If you block old addresses that are being redirected, the bot cannot see the redirect.

Testing before going live

Before changing the file, take a backup of the current version. Then follow these steps:

  1. Open the file in your browser (/robots.txt) and confirm that it loads as plain text with a 200 status code.
  2. Open the Settings → robots.txt report in Search Console; it shows the version Google last downloaded, along with syntax errors and warnings.
  3. Use the URL Inspection tool for several important pages; if you see the message "Blocked by robots.txt," review the rule.
  4. In the week after publishing, check the Pages report for an unexpected increase in the count under "Blocked by robots.txt." Our Google indexing problems guide will help you diagnose general indexing issues.

For official documentation, see the robots.txt guide on Google Search Central. robots.txt is only one item in the overall health check; for all items, use our technical SEO checklist.

Frequently asked questions

Is robots.txt mandatory?

No. If the file does not exist, bots crawl the site freely. However, if there are areas you want to block or you want to announce the sitemap address, creating one is useful.

When will a change to robots.txt take effect?

Google keeps the file in cache and usually refreshes it within a day at most; so the effect may become visible anywhere from a few hours to a day. In urgent cases, a recrawl can be requested from the Search Console robots.txt report.

Can I block AI bots with robots.txt?

Yes, for bots that follow the rules; each bot has its own user-agent name and can be written as a separate group. However, robots.txt is a request, not an enforcement mechanism; it will not stop bots that ignore the rules.

Why is a page I blocked with Disallow still appearing on Google?

Because Disallow is not a command to remove from the index. The address may be listed without content because it receives links from elsewhere. Solution: remove the block and add a noindex to the page, or remove the page entirely.

Conclusion

robots.txt is a small file with big consequences. The rule is simple: block only the areas that are genuinely worthless, do not touch CSS/JS files, add the sitemap address, and do not use it as a way to remove pages from the index. Verifying the file in Search Console after each release and making sure your important pages can be crawled prevents most disasters from the start. As your next step, you can move on to our XML sitemap guide to prepare your sitemap file.

Is your keyword missing clicks?

HitBot generates real sessions that search for your target keyword, find your site and click it. The results accumulate in Search Console as impressions and clicks.

Other posts