How to Configure Robots.txt in Magento 2

The following steps show you how to configure Robot txt in Magento 2 ?

  1. On the Admin sidebar, select Stores
  2. Develop the Search Engine Robots section
  3. When complete, choose Save Config

Hello friends,

As you know, the Magento configuration contains settings to create and control instructions for a Web Crawlers and bots that index your site. The guidelines are saved in a file called “robots.txt” that resides in the root of your Magento installation. The instructions are commands that are documented and followed by most search engines. By default, the robots.txt file that is produced by Magento consists of commands for web crawler to evade indexing assured parts of the site that contain files that are used internally by the system. You can use the default settings or describe your own custom instructions for all or for specific search engines.

Here we will discuss about how to configure robot.txt and establish about some Custom Instruction

  1. On the Admin sidebar, choose Stores

In settings, select configuration. Then, select Design in the panel on the left under General.

  1. Enlarge the Search Engine Optimization Section

The following are:

Select Default Robots

  • INDEX, FOLLOW initiates web crawlers to catalog the site and to check back later for modify.
  • NO INDEX, FOLLOW initiates web crawlers to evade indexing the site, but to check back later or modify.
  • INDEX, NO FOLLOW initiates web crawlers to index the site once, but to not check back later for modify.
  • NO INDEX, NO FOLLOW initiates web crawlers to evade indexing the site and not verify back later for modify.

After that, tap Reset to Default to restore the default instructions.

  1. When complete, Save Config

Here, we will establish some examples of Custom Instruction, the following are:

  • Allows full Access
  • Disallows access to all Folders
  • Default instructions

Allows Full Access

  • User-agent:*
  • Disallow:

Disallows Access to All Folders

  • User-agent:*
  • Disallow: /

Default Instructions

Disallow: /lib/

Disallow: /*.php

Disallow: /pkginfo/

Disallow: /var/

Disallow: /catalog/

Disallow: /customer/

Disallow: /sendfriend/

Disallow: /review/

Disallow: /*SID=

The following steps help you to learn how to Configure Robots in Magento 2

Last Update: April 9, 2018  

February 7, 2018   151   Nandini R    SEO & Search    
Total 2 Votes:
0

Tell us how can we improve this post?

+ = Verify Human or Spambot ?

Leave a Reply

Your email address will not be published. Required fields are marked *

Facebook
Google+
http://www.webnexs.com/blog/kb/configure-robots-txt-magento-2
Twitter
LinkedIn