Robots.txt

What is robots.txt?

The robots.txt file is a text document that serves as a recommendation for search engines. It allows website owners to manage which pages and sections of their resource can be indexed by search bots. This file is placed in the root directory of the site and contains directives that can either allow or prohibit the crawling of certain pages. This is crucial for ensuring proper indexing and optimization of the site to avoid exposing confidential information in search results.

What is the purpose of robots.txt?

The main purpose of using the robots.txt file is to instruct search engines on which pages to crawl and which to ignore. If these rules are not specified, crawling bots may index the site at their discretion, which can lead to undesirable consequences, such as indexing administrative pages or duplicate content. This, in turn, can negatively impact SEO optimization and the promotion of the resource, as search engines may fail to recognize important pages.

Additionally, the robots.txt file helps reduce server load by limiting the number of requests, which also positively affects the overall performance of the site. Although the file is not mandatory, many SEO specialists recommend its use as part of internal and external optimization.

How to create robots.txt?

Creating a robots.txt file does not require special skills and can be done using any text editor, such as Notepad in Windows or TextEdit in macOS. However, it is important to follow some formatting rules:

• The file name must be robots.txt;
• The file must be in text format (txt);
• The file encoding should be UTF-8.

After creating an empty file, it can be uploaded to the site. By default, it will be considered permissive. To manage indexing, specific rules for certain pages must be specified. Typically, access is restricted to pages not intended for public access, such as login pages, admin panels, and technical directories.

Directives in robots.txt

The directives in robots.txt are commands that instruct search bots on how to interact with the site. Each group of rules begins with the User-agent directive, which specifies which particular bot the following instructions are addressed to. For example:

User-agent: Googlebot

After the User-agent directive, instructions follow that may include the Disallow and Allow directives. The Disallow directive is used to prohibit indexing of certain pages or directories, while Allow permits indexing. For example, to block the entire site from indexing, the following entry can be used:

User-agent: *
Disallow: /

Special symbols in robots.txt

Special symbols can also be used in the robots.txt file to clarify the rules for search bots:

• * — a symbol that prohibits indexing of a page at the specified address;
• # — used to add comments that are ignored by bots;
• $ — placed at the end of a URL to negate the effect of the special symbol *.

These symbols help to fine-tune the file, preventing the indexing of unwanted pages and keeping important content secure.

How to upload and check robots.txt?

After creation, the robots.txt file must be uploaded to the root directory of the site. The method of uploading depends on the site's architecture and the server used. After uploading, it is important to check the file's accessibility by entering the following format in the address bar:

https://your_website.com/robots.txt

To check the functionality of the file, one can use tools provided by search engines, such as Google Search Console for Google and Yandex Webmaster for Yandex.

Common errors in configuring the robots.txt file

Some common mistakes in configuring the robots.txt file can lead to its malfunction. Here are a few of them:

• Empty User-agent directive — no indication of which bots the rules are intended for;
• Missing / or * symbol at the beginning of Disallow or Allow rules;
• Entry Disallow: / for an active site, leading to a complete block of indexing;
• Missing : between the directive and rule, making them unclear to the bots.

How do search engines understand the robots.txt file?

Search engines like Google and Yandex may interpret the robots.txt file differently. Yandex strictly adheres to the directives, and if a page is blocked by the Disallow directive, it will not be crawled. Google, on the other hand, treats the Disallow directive more as a recommendation than a prohibition. This means that to protect confidential pages, it is advisable to use additional methods, such as passwords or noindex directives.