Meta name robots tag

Meta Name Robots tag

< meta name = "robots"

How do you explain to a spider of a search engine you only want it to index just the first page of your website or that it is allowed to index the whole website. You use a specific HTML meta tag the so called the meta robots tag.

There are two main types of robots meta directives: the meta robots tag and the x-robots-tag. Any parameter that can be used in a meta robots tag can also be specified in an xrobotstag.

Key Takeaway

  1. Always add the Robots tag in the HEAD section of your site.

  2. Adding noindex and nofollow means your website won’t show up in the search results of a search engine.

  3. Robots can ignore your meta tag. Especially malware robots that scan the web for security vulnerabilities, and email address harvesters used by spammers will pay no attention.

SEO Influence Meter

0%

The robots tag can
break your website!

Overall Influence

Website SEO Score

Configure your robots tag
wrong and you’ll have no score.

The difference between the robots meta tag and the X-robots-tag

The meta robots tag allows you to control indexing behavior at page level. The x-robots-tag can be included as part of the HTTP header to control indexing of a page as a whole, as well as very specific elements of a page.

The following values (‘parameters’) can be placed on their own, or together in the content attribute of tag (separated by a comma), to control how search engines interact with your page.

index or noindex
follow or nofollow
noimageindex
noarchive
nocache
nosnippet
notranslate
noyaca

To use the x-robots-tag, you’ll need to have access to either your website’s header .php, .htaccess, or server access file.
When to use the x-robot-tag:

  • Controlling the indexation of content not written in HTML (like flash or video)
  • Blocking indexation of a particular element of a page (like an image or video). But not of the entire page itself
  • If you don’t have access to a page’s HTML (specifically, to the <head> section). Or if your site uses a global header that cannot be changed
  • You do not need to use both meta robots and the x-robots-tag on the same page – doing so would be redundant.

The Robots Meta tag has four main values for search engine crawlers:

  • FOLLOW – The search engine crawler will follow all the links in that web page
  • INDEX – The search engine crawler will index the whole web page
  • NOFOLLOW – The search engine crawler will NOT follow the page and any links in that web page
  • NOINDEX – The search engine crawler will NOT index that web page

Other values are:

  • NOIMAGEINDEX – Disallow search engines from indexing images on the page
  • NOARCHIVE – Prevents the search engines from showing a cached copy of this page in their search results listings
  • NOCACHE – Same as noarchive but only used by MSN/Live
  • NOSNIPPET – Prevents the search engines from showing a text or video snippet of this page in the search results, and prevents them from showing a cached copy of this page in their search results listings
  • NOTRANSLATE – Prevents search engines from showing translations of the page in their search results
  • NOCAYA – Prevents the search results snippet from using the page description from the Yandex Directory

 

HTML Code Robots Example

Not all pages of your WordPress website need to be indexed by the search engines. That is why Yoast gives you the possibility to  decide which pages can or can’t be indexed.

The Yoast SEO plugin has a special checkbox, also knows as the “Noindex subpages of archives“. It is easier to add metatags by using this checkbox.  Go to Advanced features and select noindex in the empty field right after Meta Robots Index.

 

TIP: Working with WordPress?

Check our WordPress Metatags Manual

Select the robots metatag by using the WordPress Yoast checkbox

Affordable & Effective

We offer affordable SEO services that are fast and effective.

Need some professional
SEO help?

Contact our Customer Service representative to make an appointment and to answer all your questions!

SEO & Online Marketing blog

Google Rethinking Payday Loans & Doorway Pages?

Google Rethinking Payday Loans & Doorway Pages?

Best yet, not only does Google maintain their investment in payday loans via LendUp, but there is also a bubble…
How Google Search Works in 2017

How Google Search Works in 2017

Many years ago we created an infographic about how search works, from the perspective of a content creator, starting with…
Benefits of Using Modern Promotion Technologies

Benefits of Using Modern Promotion Technologies

Technology is constantly changing, and it is obvious it takes our life to a whole new level. As the Internet…