What Is robots.txt?

robots.txt is a plain text file located in the root directory of your website that tells search engine bots (such as Googlebot and Bingbot) which parts of your site they may crawl and which parts they should not. You can reach any site's robots.txt by typing https://www.yoursite.com/robots.txt into the address bar.

You can think of this file as a "doorkeeper": it guides bots as a matter of courtesy. There is, however, an important point to remember; robots.txt is not a security tool. Blocking a page here does not mean it will never appear in Google, nor does it guarantee that the page stays hidden.

What Does robots.txt Do?

The main purpose of robots.txt is to help search engines use their crawl budget efficiently. By closing off areas that have no SEO value — such as admin panels, search result pages, cart and checkout pages — to crawling, you let bots focus on the content that actually matters. You can also help search engines easily find your map by specifying the address of your sitemap in this file.

Basic robots.txt Commands

The file consists of simple commands:

  • User-agent: Specifies which bot the commands apply to. The * sign means "all bots".
  • Disallow: Prevents the specified path from being crawled.
  • Allow: Permits a specific path inside a blocked folder.
  • Sitemap: Declares the full address of your sitemap.

Example robots.txt Files

The recommended structure for a standard WordPress site:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://www.siteniz.com/sitemap_index.xml

This structure closes the admin panel to crawling while allowing the admin-ajax.php file that themes and plugins need to work properly.

The simplest structure, opening the entire site to crawling:

User-agent: *
Disallow:

The empty Disallow command here means "do not block anything".

Caution — the dangerous structure that blocks the entire site:

User-agent: *
Disallow: /

A single / sign blocks your entire site from being crawled. When this line is added by mistake, it can drop your site completely out of Google; so be very careful.

How to Create robots.txt

In WordPress

WordPress generates a virtual robots.txt by default. You can view and edit this file from the "Tools > File Editor" section of the Yoast SEO or Rank Math plugin. The plugin saves the changes for you and places them in the correct location.

Creating It Manually

Open a text editor such as Notepad, type the commands, and save the file under the name robots.txt (with a .txt extension). Then upload it to the root directory of your site (public_html) via FTP or your hosting file manager. The file must be in the root; a robots.txt in a subfolder is considered invalid.

Testing robots.txt

After publishing your file, check your syntax with the robots.txt Test Tool in Google Search Console. This tool shows you faulty commands and important pages you have accidentally blocked. We strongly recommend testing before going live, because even a small typo can drop your entire site from search results.

robots.txt and Its Relationship to SEO

The role of robots.txt in SEO is often misunderstood. This file does not directly improve your rankings; its real function is to manage crawl budget. Google allocates a limited crawling resource to each site. If bots spend their time on worthless addresses such as filtered search results, endless calendar pages, or admin areas, your truly important content is crawled later. A properly configured robots.txt directs bots to valuable pages and thereby provides an indirect SEO advantage.

This is especially critical for large e-commerce sites. The countless parameterized URLs created by color, size, and sorting filters can quickly consume the crawl budget. Managing these kinds of parameterized addresses with robots.txt ensures your site is crawled efficiently.

Advanced Usage Examples

Beyond the basic commands, there are some useful patterns. You can use a wildcard to block a specific file type:

User-agent: *
Disallow: /*.pdf$

This command blocks all PDF files from being crawled. To block addresses containing a specific parameter:

Disallow: /*?filtre=

You can also define different rules for different bots. For example, if you want to block only an image bot, you target the relevant "User-agent" line. But remember: rules are applied not from top to bottom, but according to the most specific match, and a single mistake can affect your entire site. That is why you should always try advanced rules in the test tool before going live.

Things to Watch Out For When Managing robots.txt

robots.txt is a powerful file, and when used correctly it ensures your site is crawled efficiently; however, a single wrong line can jeopardize your entire visibility. For this reason, you need to follow a few basic rules: always keep the file in the root directory, clearly specify the paths you want to block, do not forget to add your sitemap address, and always verify with the test tool after every change. Also keep in mind that if you want a blocked page to disappear completely from search, you should use the noindex tag instead of robots.txt.

Frequently Asked Questions (FAQ)

Is robots.txt a security tool?

No. robots.txt only guides bots as a matter of courtesy and is not a security tool. Blocking a page here does not guarantee that it will stay hidden or never appear in Google. Real protection requires methods such as a password or noindex.

Where should the robots.txt file be located?

The file must be in your site's root directory (public_html) and must be accessible at https://www.yoursite.com/robots.txt. A robots.txt in a subfolder is considered invalid.

Does robots.txt improve my ranking?

Not directly. Its real function is to manage crawl budget. By steering bots away from worthless pages and toward valuable content, it provides an indirect SEO advantage.

Conclusion

robots.txt is a small but critical file that guides search engine bots correctly across your site. When configured properly it focuses your crawl budget on valuable pages; a single wrong line, however, can drop your entire site from search. Keep the commands simple, add your sitemap, and never skip testing after every change.

Related Articles