Robot control with <meta> tag

In addition to controlling robots access to your website with a robots.txt file, you can use the <meta> tag with the robots attribute. If you use the <meta> tag method to restrict robots' access, it must be placed in the page that you do not want indexed. The second attribute that needs to be used with the <meta> tag is content. The content attribute determines what you want the robot to do for that particular webpage:

  • all — Index the page and follow the links. If the robot does not see any meta tag with robots attribute, it indexes the current page and follows all links.
  • none — Do not index the page and do not follow any links on the page.
  • index — Index the page
  • nonindex — Do not index the page
  • follow — Follow all the links on the page
  • nofollow — Do not follow all the links on the page.

The following, for example,

<meta name="robots" content="index">
<meta name="robots" content="nofollow">

says that index the page but do not follow any links on the page. You could combine the top two tags as one to achieve the same result:

<meta name="robots" content="index, nofollow">

When using the <meta> tag to exclude robots to a particular page, make sure not to specify contradictory instruction such as:

<meta name="robots" content="noindex, index">

or

<meta name="robots" content="follow, nofollow">

Any one of these instructions may be ignored completely by the spiders or may be processed partially. Also, the other disadvantage of using the <meta> tag approach is that it is not supported as widely as the robots.txt file.