{"id":3052,"date":"2020-07-18T09:11:54","date_gmt":"2020-07-18T06:11:54","guid":{"rendered":"https:\/\/esfirum.com\/?p=3052"},"modified":"2021-04-20T18:14:54","modified_gmt":"2021-04-20T15:14:54","slug":"robots-txt-dlya-wordpress","status":"publish","type":"post","link":"https:\/\/esfirum.com\/en\/blog\/robots-txt-dlya-wordpress\/","title":{"rendered":"How to Optimize WordPress Robots.txt File"},"content":{"rendered":"<p>Webmasters and marketers know how important the indexing of a site by search engines is. This is why they do everything they can to help search engines like Google and Yandex crawl and index their sites correctly.<br \/>\nA lot of time and resources are spent on internal and external optimization such as content, links, tags, image optimization and site structure.<br \/>\nAll this plays a huge role in promotion. However, if you forgot to do the technical optimization of the site, if you have not heard about the robots.txt and sitemap.xml files, there may be problems with the correct crawling and indexing of your site.<br \/>\nIn this article, I will explain how to properly configure and use a robots.txt file and a robots meta tag. So, let&#8217;s begin!<\/p>\n<h2>What robots.txt file is?<\/h2>\n<p><strong>Robots.txt<\/strong> &#8211; it is a text file that is used as a guide for search engine robots (also known as crawlers, bots, or spiders) how to crawl and index pages on a site.<\/p>\n<p>In simple terms, robots.txt tells robots which pages or site files we want to see in search and which we don&#8217;t.<\/p>\n<p>Ideally, the robots.txt file is placed in the root directory of your website (https:\/\/site.com\/robots.txt) so that robots can immediately access its instructions.<\/p>\n<p>If you are using CMS WordPress, then you will be able to see your file at the above address, however you will not find the file itself in the shared folder with your site. This is because WordPress automatically creates a virtual robots.txt file (with default parameters) if it does not find the file in the site&#8217;s root directory.<\/p>\n<p>The virtual robots.txt file of the CMS WordPress does not solve all the necessary tasks, so it is highly desirable to write your own.<\/p>\n<h2>What is robots.txt for?<\/h2>\n<p>The robots.txt file is needed to prevent search robots from visiting certain sections of your site, for example:<\/p>\n<ul>\n<li>pagination pages;<\/li>\n<li>pages with search results on the site;<\/li>\n<li>administrative files;<\/li>\n<li>service pages;<\/li>\n<li>links with utm tags;<\/li>\n<li>data on sorting, filtering, comparison parameters;<\/li>\n<li>personal account page, etc.<\/li>\n<\/ul>\n<p><strong>Important!<\/strong> The robots.txt file is optional for search robots. In this regard, if you want to be 100% sure that any of the pages of your site will not appear in the search results, use the robots meta tag.<br \/>\nAccording to <a href=\"https:\/\/developers.google.com\/search\/docs\/advanced\/robots\/intro\" target=\"_blank\" rel=\"noopener noreferrer\">Google Help<\/a> <strong>\u00a0the robots.txt file is not intended to prevent web pages from showing in Google search results.<br \/>\n<\/strong>If you do not want any page of your site to appear in the search, insert into page &lt;head&gt;\u00a0attribute noindex:<\/p>\n<pre class=\"lang:default decode:true \">&lt;meta name=\u201crobots\u201d content=\u201cnoindex,nofollow\u201d&gt;<\/pre>\n<h2>How to edit robots.txt<\/h2>\n<p>There are two ways to edit your robots.txt file in WordPress CMS. Add the required code to the functions.php file, or using a plugin.<br \/>\nIn our agency, we prefer the second way.<br \/>\nInstall the plugin <a href=\"https:\/\/wordpress.org\/plugins\/pc-robotstxt\/\" target=\"_blank\" rel=\"nofollow noopener noreferrer\">Virtual Robots.txt<\/a> from the CMS WordPress repository, open it in the admin. panel in the Settings tab. In the plugin field that opens, enter the required code, press the Save button and voil\u00e0 &#8211; your robots.txt file is ready.<\/p>\n<h2>Proper robots.txt for WordPress CMS<\/h2>\n<pre class=\"lang:default decode:true\">User-agent: *               # general rules for robots of all search engines\r\nDisallow: \/cgi-bin          # service folder for storing server scripts\r\nDisallow: \/?                # all request parameters on the main\r\nDisallow: \/wp-              # all files WP: \/wp-json\/, \/wp-includes, \/wp-content\/plugins\r\nDisallow: \/wp\/              # if there is a subdirectory \/ wp \/ where the CMS is installed (if not,\r\n                            # rule can be removed)\r\nDisallow: *?s=              # site search\r\nDisallow: *&amp;s=          # site search\r\nDisallow: \/search\/          # site search\r\nDisallow: \/author\/          # author's archive\r\nDisallow: \/users\/           # user archive\r\nDisallow: *\/trackback       # trackbacks, notifications in comments about a link to a web document\r\nDisallow: *\/feed            # all feeds\r\nDisallow: *\/rss             # rss feed\r\nDisallow: *\/embed           # all embeddings\r\nDisallow: *\/wlwmanifest.xml # xml Windows Live Writer manifest file (if not using,\r\n                            # rule can be removed)\r\nDisallow: \/xmlrpc.php       # WordPress API file\r\nDisallow: *utm*=            # utm\r\nDisallow: *openstat=        # openstat tags\r\nAllow: *\/uploads            # open the folder with files uploads\r\nAllow: \/*\/*.js              # open js files\r\nAllow: \/*\/*.css             # open css files\r\nAllow: \/wp-*.png            # allow indexing images\r\nAllow: \/wp-*.jpg            # allow indexing images\r\nAllow: \/wp-*.jpeg           # allow indexing images\r\nAllow: \/wp-*.gif            # allow indexing gifs\r\nAllow: \/wp-admin\/admin-ajax.php # allow ajax\r\n\r\n# Specify one or more Sitemap.xml files. Google XML Sitemap Plugin automatically creates 2 sitemaps like in the example below.\r\nSitemap: http:\/\/yoursite.\/sitemap.xml\r\nSitemap: http:\/\/yoursite.ru\/sitemap.xml.gz\r\n<\/pre>\n<h2>How to check robots.txt file?<\/h2>\n<p>If the robots.txt file is not configured correctly, it can lead to multiple errors in the indexing of the site. You can check the correctness of your robots.txt settings using a free tool <a href=\"https:\/\/www.google.com\/webmasters\/tools\/robots-testing-tool\" target=\"_blank\" rel=\"noopener noreferrer\">Google Robots Testing Tool<\/a><\/p>\n<p>Choosing your site:<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter wp-image-3412 size-large\" src=\"https:\/\/esfirum.com\/wp-content\/uploads\/2020\/07\/robots-txt-cheker-1024x488.jpg\" alt=\"robots.txt checker\" width=\"1024\" height=\"488\" srcset=\"https:\/\/esfirum.com\/wp-content\/uploads\/2020\/07\/robots-txt-cheker-1024x488.jpg 1024w, https:\/\/esfirum.com\/wp-content\/uploads\/2020\/07\/robots-txt-cheker-300x143.jpg 300w, https:\/\/esfirum.com\/wp-content\/uploads\/2020\/07\/robots-txt-cheker-768x366.jpg 768w, https:\/\/esfirum.com\/wp-content\/uploads\/2020\/07\/robots-txt-cheker-1536x732.jpg 1536w, https:\/\/esfirum.com\/wp-content\/uploads\/2020\/07\/robots-txt-cheker.jpg 1900w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/p>\n<p>As a result, there should be no errors or warnings and the file should be Accessible for robots:<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter wp-image-3411 size-large\" src=\"https:\/\/esfirum.com\/wp-content\/uploads\/2020\/07\/robots-txt-checker-result-1024x487.jpg\" alt=\"robots.txt checker result\" width=\"1024\" height=\"487\" srcset=\"https:\/\/esfirum.com\/wp-content\/uploads\/2020\/07\/robots-txt-checker-result-1024x487.jpg 1024w, https:\/\/esfirum.com\/wp-content\/uploads\/2020\/07\/robots-txt-checker-result-300x143.jpg 300w, https:\/\/esfirum.com\/wp-content\/uploads\/2020\/07\/robots-txt-checker-result-768x365.jpg 768w, https:\/\/esfirum.com\/wp-content\/uploads\/2020\/07\/robots-txt-checker-result-1536x730.jpg 1536w, https:\/\/esfirum.com\/wp-content\/uploads\/2020\/07\/robots-txt-checker-result.jpg 1898w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/p>\n<p>&nbsp;<\/p>\n<p>If your robots.txt file is configured correctly, it will significantly speed up the indexing process of your site.<\/p>","protected":false},"excerpt":{"rendered":"<p>Webmasters and marketers know how important the indexing of a site by search engines is. This is why they do <span class=\"view-full-post\"><a href=\"https:\/\/esfirum.com\/en\/blog\/robots-txt-dlya-wordpress\/\" class=\"view-full-post-btn\">&#8230;<\/a><\/span><\/p>\n","protected":false},"author":4,"featured_media":3264,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[17],"tags":[],"class_list":["post-3052","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-blog"],"acf":[],"_links":{"self":[{"href":"https:\/\/esfirum.com\/en\/wp-json\/wp\/v2\/posts\/3052","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/esfirum.com\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/esfirum.com\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/esfirum.com\/en\/wp-json\/wp\/v2\/users\/4"}],"replies":[{"embeddable":true,"href":"https:\/\/esfirum.com\/en\/wp-json\/wp\/v2\/comments?post=3052"}],"version-history":[{"count":0,"href":"https:\/\/esfirum.com\/en\/wp-json\/wp\/v2\/posts\/3052\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/esfirum.com\/en\/wp-json\/wp\/v2\/media\/3264"}],"wp:attachment":[{"href":"https:\/\/esfirum.com\/en\/wp-json\/wp\/v2\/media?parent=3052"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/esfirum.com\/en\/wp-json\/wp\/v2\/categories?post=3052"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/esfirum.com\/en\/wp-json\/wp\/v2\/tags?post=3052"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}