Put simply, robots.txt tells the robots which pages or files of the website we want to see in search and which we do not.<\/p>\n
Ideally the robots.txt file is placed in the root directory of your website (https:\/\/site.com\/robots.txt) so that the robots can access its instructions straight away.<\/p>\n
If you use CMS WordPress you will be able to see your file at the address above, but you will not find the file itself in the general folder with your website. That is because WordPress automatically creates a virtual robots.txt file (with default parameters) if it does not find this file in the root directory of the site.<\/p>\n
The virtual robots.txt file of CMS WordPress does not solve all the necessary tasks, so it is highly advisable to write your own.<\/p>\n
Why you need robots.txt<\/h2>\n
The robots.txt file is needed in order to stop search robots visiting certain sections of your site, for example:<\/p>\n
- \n
- pagination pages;<\/li>\n
- pages with search results on the site;<\/li>\n
- administrative files;<\/li>\n
- service pages;<\/li>\n
- links with utm tags;<\/li>\n
- data about sorting, filtering and comparison parameters;<\/li>\n
- the personal account page and so on.<\/li>\n<\/ul>\n
Important!<\/strong> The robots.txt file is not binding for search robots. Because of that, if you want to be 100% sure that some page of your website will not appear in the search results - use the robots meta tag.
\nAccording to the Google help page<\/a> the robots.txt file is not meant to keep web pages out of Google search results.
\nIf you do not want a page of your website to appear in search, add the noindex attribute in the page :<\/p>\n<meta name=\u201crobots\u201d content=\u201cnoindex,nofollow\u201d><\/pre>\n
How to edit robots txt<\/p>\n
There are two ways to edit the robots.txt file in CMS WordPress. Add the necessary code to the functions.php file, or use a plugin.
\nIn our company we prefer the second method.
\nWe install the plugin Virtual Robots.txt<\/a> from the CMS WordPress repository, we open it in the admin panel in the Settings tab. In the plugin field that opens we add the necessary code, press Save and voila - your robots.txt file is ready.<\/p>\nThe right robots.txt for CMS WordPress<\/p>\n
Taken from the website seogio.ru<\/a> and adjusted it a little. Here is the result:<\/p>\n
User-agent: * # general rules for the robots of all search engines Disallow: \/cgi-bin # service folder for storing server scripts Disallow: \/? # all query parameters on the home page Disallow: \/wp- # all WP files: \/wp-json\/, \/wp-includes, \/wp-content\/plugins Disallow: \/wp\/ # if there is a \/wp\/ subdirectory where the CMS is installed (if not, the rule can be removed) Disallow: *?s= # site search Disallow: *&s= # site search Disallow: \/search\/ # site search Disallow: \/author\/ # author archive Disallow: \/users\/ # user archive Disallow: *\/trackback # trackbacks, notifications in comments about a link to a web document Disallow: *\/feed # all feeds Disallow: *\/rss # rss feed Disallow: *\/embed # all embeds Disallow: *\/wlwmanifest.xml # the Windows Live Writer manifest xml file (if you do not use it, the rule can be removed) Disallow: \/xmlrpc.php # the WordPress API file Disallow: *utm*= # links with utm tags Disallow: *openstat= # links with openstat tags Allow: *\/uploads # we open the folder with the uploads files Allow: \/*\/*.js # we open the js script files Allow: \/*\/*.css # we open the css files Allow: \/wp-*.png # we allow images to be indexed Allow: \/wp-*.jpg # we allow images to be indexed Allow: \/wp-*.jpeg # we allow images to be indexed Allow: \/wp-*.gif # we allow gifs to be indexed Allow: \/wp-admin\/admin-ajax.php # we allow ajax # Specify the main mirror of the site, as in the example below (with WWW \/ without WWW, and if HTTPS # then we write the protocol; if you need to specify a port, specify it). The directive has become optional. Host used to be understood by # Yandex and Mail.RU. Now none of the major search engines take the Host directive into account. Host: www.site.ru # Specify one or several Sitemap files (there is no need to duplicate it for every User-agent). Google XML Sitemap creates 2 sitemaps, as in the example below. Sitemap: http:\/\/site.ru\/sitemap.xml Sitemap: http:\/\/site.ru\/sitemap.xml.gz\r\n\r\n<\/pre>\n
Checking robots.txt<\/h2>\n
If the robots.txt file is configured incorrectly it can lead to multiple website indexing errors.
\nYou can check whether robots.txt is configured correctly with a free tool Google Robots Testing Tool<\/a><\/p>\nWe choose our website:
\n![\"robots](\"https:\/\/esfirum.com\/wp-content\/uploads\/2020\/06\/gsc-robots-txt-1-1024x569.jpg\")
<\/p>\nWe type the path to our robots.txt file into the field and press the Check button:<\/p>\n
![\"robots](\"https:\/\/esfirum.com\/wp-content\/uploads\/2020\/06\/gsk-robots-txt-2-1024x56.jpg\")
<\/p>\nAs a result there should be no errors or warnings and the file should be Available to robots:<\/p>\n
![\"robots](\"https:\/\/esfirum.com\/wp-content\/uploads\/2020\/06\/gsc-robots-txt-check-1024x408.jpg\")
<\/p>\n<\/p>\n
If the robots.txt file is configured correctly, this will speed up the indexing of your site considerably.<\/p>","protected":false},"excerpt":{"rendered":"
\u0412\u0435\u0431\u043c\u0430\u0439\u0441\u0442\u0440\u0438 \u0442\u0430 \u043c\u0430\u0440\u043a\u0435\u0442\u043e\u043b\u043e\u0433\u0438 \u0437\u043d\u0430\u044e\u0442\u044c \u043d\u0430\u0441\u043a\u0456\u043b\u044c\u043a\u0438 \u0432\u0430\u0436\u043b\u0438\u0432\u0430 \u0456\u043d\u0434\u0435\u043a\u0441\u0430\u0446\u0456\u044f \u0441\u0430\u0439\u0442\u0443 \u043f\u043e\u0448\u0443\u043a\u043e\u0432\u0438\u043c\u0438 \u0441\u0438\u0441\u0442\u0435\u043c\u0430\u043c\u0438. \u0421\u0430\u043c\u0435 \u0442\u043e\u043c\u0443 \u0432\u043e\u043d\u0438 \u0440\u043e\u0431\u043b\u044f\u0442\u044c \u0432\u0441\u0435 \u043c\u043e\u0436\u043b\u0438\u0432\u0435, \u0449\u043e\u0431 \u0434\u043e\u043f\u043e\u043c\u043e\u0433\u0442\u0438 \u043f\u043e\u0448\u0443\u043a\u043e\u0432\u0438\u043a\u0430\u043c \u0442\u0438\u043f\u0443 …<\/a><\/span><\/p>","protected":false},"author":4,"featured_media":3264,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[17],"tags":[],"class_list":["post-3052","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-blog"],"acf":[],"_links":{"self":[{"href":"https:\/\/esfirum.com\/en\/wp-json\/wp\/v2\/posts\/3052","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/esfirum.com\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/esfirum.com\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/esfirum.com\/en\/wp-json\/wp\/v2\/users\/4"}],"replies":[{"embeddable":true,"href":"https:\/\/esfirum.com\/en\/wp-json\/wp\/v2\/comments?post=3052"}],"version-history":[{"count":0,"href":"https:\/\/esfirum.com\/en\/wp-json\/wp\/v2\/posts\/3052\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/esfirum.com\/en\/wp-json\/wp\/v2\/media\/3264"}],"wp:attachment":[{"href":"https:\/\/esfirum.com\/en\/wp-json\/wp\/v2\/media?parent=3052"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/esfirum.com\/en\/wp-json\/wp\/v2\/categories?post=3052"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/esfirum.com\/en\/wp-json\/wp\/v2\/tags?post=3052"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}