Learn SEO Icon

The Meta Robots Tag

The Meta Robots tag should ONLY be used to instruct search engines to do what they do not do by default. Let us show you how and when to use the Meta Robots tag.

All major search engines support the <meta> Robots tag. With this page we will concentrate on those directives that all search engine crawlers support.

The Meta Robots Element

The <meta> Robots tag, often referred to as the robots exclusion standard or the robots exclusion protocol is a meta tag standard used by website owners to interrupt the normal behavior of crawler based search engines. If you do not want to restrict search engines access to your web pages, then you have no need to use the Meta Robots tag. If however, you want better control over the way search engines visit and index individual web pages, then read on.

When used, the <meta> Robots tag is found in your HTML source code as highlighted below:

HTML Meta Robots Tag Example

<!DOCTYPE html>
<html>
<head>
  <meta charset="utf-8">
  <title>Example Page</title>
  <meta name="robots" content="NOARCHIVE">
  <meta name="description" content="This is an example page to help us better understand how search engines see our web page.">
  <meta name="keywords" content="example page, search engines, human visitors">
</head>
<body>
  <h1>Example Page</h1>
  <p>This is a <b>example page</b> to help us better understand how <b>search engines</b> and <b>human visitors</b> see our web page.</p>
  </body>
</html>
Tip: Just because your Meta Robots tag is in your HTML source code does NOT mean you have to edit your HTML source code to add or edit one. Every web page authoring program allows you to easily edit your Meta Robots tag without you editing the HTML source code.

How Does the Meta Robots Work?

By default, all major search engines are looking for unique content to include in their search results. Therefore, when a search engine discovers a new page, they will "by default" index the page (capture all the words) and will follow all the links on the page so it can discover more pages. That said, you can use the Meta Robots tag to fine tune the way search engine crawlers are allowed to access and index the content of your web pages. Below is a list of special directives which all major search engine crawlers support. You can combine multiple directives using a comma (e.g. "noindex,nofollow").

Directive Meaning
NOINDEX Will prevent "friendly" robots from indexing the page. However, robots are still allowed to follow links found on the page.
NOFOLLOW Prevents "friendly" robots from following links found on the page.
NOARCHIVE Prevents some search engines from storing a cached copy of your page.
NOSNIPPET Will prevent Bing, Google and others from showing a snippet (description) nor a preview thumbnail (were applicable) for the page.
NOODP Will prevent Bing, Google and others from using the  Open Directory Project (ODP) listing's title and/or description (also called snippet) in their search results pages.
NONE This prevents anything on the page from being indexed or followed. The value of NONE is equivalent to using noindex, nofollow. Example:

<meta name="robots" content="noindex,nofollow">

Caution! The Meta Robots tag is not a secure way to prevent crawlers from indexing your content. In fact, malicious crawlers will most likely ignore your Meta Robots tag anyway.

When would I use the Meta Robots?

Generally you would not want to alter the default behavior of search engine crawlers, but there are times when doing so makes sense. For example, we include the following Meta Robots tag on most pages:

<meta name="robots" content="noodp">

The reason we include this is because the title and description the  DMOZ has for some of our pages is decades old and no longer relevant. Using the Meta Robots tag is your choice and generally not necessary.