Robot control with <meta> tag

In addition to controlling robots access to your website with a robots.txt file, you can use the <meta> tag with the robots attribute. If you use the <meta> tag method to restrict robots' access, it must be placed in the page that you do not want indexed. The second attribute that needs to be used with the <meta> tag is content. The content attribute determines what you want the robot to do for that particular webpage:

• all — Index the page and follow the links. If the robot does not see any meta tag with robots attribute, it indexes the current page and follows all links.
• none — Do not index the page and do not follow any links on the page.
• index — Index the page
• nonindex — Do not index the page
• follow — Follow all the links on the page
• nofollow — Do not follow all the links on the page.

The following, for example,

says that index the page but do not follow any links on the page. You could combine the top two tags as one to achieve the same result:

When using the <meta> tag to exclude robots to a particular page, make sure not to specify contradictory instruction such as:

or

Any one of these instructions may be ignored completely by the spiders or may be processed partially. Also, the other disadvantage of using the <meta> tag approach is that it is not supported as widely as the robots.txt file.