How to Optimize WordPress Robots.txt File

Webmasters and marketers know how important the indexing of a website by search engines is. That is exactly why they do everything possible to help search engines like Google and Yandex crawl and index their sites correctly.
A great deal of time and resources is spent on on-page and off-page optimisation, such as content, links, tags, image optimisation and website structure.
All of this plays a huge role in promotion. However, if you forgot to do technical optimisation of the website, if you have not heard about the robots.txt and sitemap.xml files, problems may arise with the correct crawling and indexing of your site.
In this article I will explain how to set up and use the robots.txt file and the robots meta tag correctly. So let's begin!

What is robots.txt

Robots.txt - is a text file used as an instruction for search engine robots (also known as crawlers, bots or spiders) on how to crawl and index the pages of a website.

Put simply, robots.txt tells the robots which pages or files of the website we want to see in search and which we do not.

Ideally the robots.txt file is placed in the root directory of your website (https://site.com/robots.txt) so that the robots can access its instructions straight away.

If you use CMS WordPress you will be able to see your file at the address above, but you will not find the file itself in the general folder with your website. That is because WordPress automatically creates a virtual robots.txt file (with default parameters) if it does not find this file in the root directory of the site.

The virtual robots.txt file of CMS WordPress does not solve all the necessary tasks, so it is highly advisable to write your own.

Why you need robots.txt

The robots.txt file is needed in order to stop search robots visiting certain sections of your site, for example:

• pagination pages;
• pages with search results on the site;
• administrative files;
• service pages;
• links with utm tags;
• data about sorting, filtering and comparison parameters;
• the personal account page and so on.

Important! The robots.txt file is not binding for search robots. Because of that, if you want to be 100% sure that some page of your website will not appear in the search results - use the robots meta tag.
According to the Google help page the robots.txt file is not meant to keep web pages out of Google search results.
If you do not want a page of your website to appear in search, add the noindex attribute in the page :

<meta name=“robots” content=“noindex,nofollow”>

How to edit robots txt

There are two ways to edit the robots.txt file in CMS WordPress. Add the necessary code to the functions.php file, or use a plugin.
In our company we prefer the second method.
We install the plugin Virtual Robots.txt from the CMS WordPress repository, we open it in the admin panel in the Settings tab. In the plugin field that opens we add the necessary code, press Save and voila - your robots.txt file is ready.

The right robots.txt for CMS WordPress

Taken from the website seogio.ru and adjusted it a little. Here is the result:

User-agent: * # general rules for the robots of all search engines Disallow: /cgi-bin # service folder for storing server scripts Disallow: /? # all query parameters on the home page Disallow: /wp- # all WP files: /wp-json/, /wp-includes, /wp-content/plugins Disallow: /wp/ # if there is a /wp/ subdirectory where the CMS is installed (if not, the rule can be removed) Disallow: *?s= # site search Disallow: *&s= # site search Disallow: /search/ # site search Disallow: /author/ # author archive Disallow: /users/ # user archive Disallow: */trackback # trackbacks, notifications in comments about a link to a web document Disallow: */feed # all feeds Disallow: */rss # rss feed Disallow: */embed # all embeds Disallow: */wlwmanifest.xml # the Windows Live Writer manifest xml file (if you do not use it, the rule can be removed) Disallow: /xmlrpc.php # the WordPress API file Disallow: *utm*= # links with utm tags Disallow: *openstat= # links with openstat tags Allow: */uploads # we open the folder with the uploads files Allow: /*/*.js # we open the js script files Allow: /*/*.css # we open the css files Allow: /wp-*.png # we allow images to be indexed Allow: /wp-*.jpg # we allow images to be indexed Allow: /wp-*.jpeg # we allow images to be indexed Allow: /wp-*.gif # we allow gifs to be indexed Allow: /wp-admin/admin-ajax.php # we allow ajax # Specify the main mirror of the site, as in the example below (with WWW / without WWW, and if HTTPS # then we write the protocol; if you need to specify a port, specify it). The directive has become optional. Host used to be understood by # Yandex and Mail.RU. Now none of the major search engines take the Host directive into account. Host: www.site.ru # Specify one or several Sitemap files (there is no need to duplicate it for every User-agent). Google XML Sitemap creates 2 sitemaps, as in the example below. Sitemap: http://site.ru/sitemap.xml Sitemap: http://site.ru/sitemap.xml.gz

Checking robots.txt

If the robots.txt file is configured incorrectly it can lead to multiple website indexing errors.
You can check whether robots.txt is configured correctly with a free tool Google Robots Testing Tool

We choose our website:

We type the path to our robots.txt file into the field and press the Check button:

As a result there should be no errors or warnings and the file should be Available to robots:

If the robots.txt file is configured correctly, this will speed up the indexing of your site considerably.