<meta> Robots tag, often referred to as the robots exclusion standard or the robots exclusion protocol is a meta tag standard used by website owners to interrupt the normal behavior of crawler based search engines. If you do not want to restrict search engines access to your web pages, then you have no need to use the Meta Robots tag. If however, you want better control over the way search engines visit and index individual web pages, then read on.
When used, the
<meta> Robots tag is found in your HTML source code as highlighted below:
<!DOCTYPE html> <html> <head> <meta charset="utf-8"> <title>Example Page</title> <meta name="robots" content="NOARCHIVE"> <meta name="description" content="This is an example page to help us better understand how search engines see our web page."> <meta name="keywords" content="example page, search engines, human visitors"> </head> <body> <h1>Example Page</h1> <p>This is a <b>example page</b> to help us better understand how <b>search engines</b> and <b>human visitors</b> see our web page.</p> </body> </html>
By default, all major search engines are looking for unique content to include in their search results. Therefore, when a search engine discovers a new page, they will "by default" index the page (capture all the words) and will follow all the links on the page so it can discover more pages. That said, you can use the Meta Robots tag to fine tune the way search engine crawlers are allowed to access and index the content of your web pages. Below is a list of special directives which all major search engine crawlers support. You can combine multiple directives using a comma (e.g. "noindex,nofollow").
|NOINDEX||Will prevent "friendly" robots from indexing the page. However, robots are still allowed to follow links found on the page.|
|NOFOLLOW||Prevents "friendly" robots from following links found on the page.|
|NOARCHIVE||Prevents some search engines from storing a cached copy of your page.|
|NOSNIPPET||Will prevent Bing, Google and others from showing a snippet (description) nor a preview thumbnail (were applicable) for the page.|
|NOODP||Will prevent Bing, Google and others from using the Open Directory Project (ODP) listing's title and/or description (also called snippet) in their search results pages.|
|NONE||This prevents anything on the page from being indexed or followed. The value of NONE is equivalent to using noindex, nofollow. Example:
<meta name="robots" content="noindex,nofollow">
Caution! The Meta Robots tag is not a secure way to prevent crawlers from indexing your content. In fact, malicious crawlers will most likely ignore your Meta Robots tag anyway.
Generally you would not want to alter the default behavior of search engine crawlers, but there are times when doing so makes sense. For example, we include the following Meta Robots tag on most pages:
<meta name="robots" content="noodp">
The reason we include this is because the title and description the DMOZ has for some of our pages is decades old and no longer relevant. Using the Meta Robots tag is your choice and generally not necessary.