Meta Name Robots tag
< meta name = "robots"
How do you explain to a spider of a search engine you only want it to index just the first page of your website or that it is allowed to index the whole website. You use a specific HTML meta tag the so called the meta robots tag.
There are two main types of robots meta directives: the meta robots tag and the x-robots-tag. Any parameter that can be used in a meta robots tag can also be specified in an x–robots–tag.
Key Takeaway
- Always add the Robots tag in the HEAD section of your site.
- Adding noindex and nofollow means your website won’t show up in the search results of a search engine.
- Robots can ignore your meta tag. Especially malware robots that scan the web for security vulnerabilities, and email address harvesters used by spammers will pay no attention.
SEO Influence Meter
The robots tag can
break your website!
Overall Influence
Website SEO Score
Configure your robots tag
wrong and you’ll have no score.
The difference between the robots meta tag and the X-robots-tag
The meta robots tag allows you to control indexing behavior at page level. The x-robots-tag can be included as part of the HTTP header to control indexing of a page as a whole, as well as very specific elements of a page.
The following values (‘parameters’) can be placed on their own, or together in the content attribute of tag (separated by a comma), to control how search engines interact with your page.
index or noindex
follow or nofollow
noimageindex
noarchive
nocache
nosnippet
notranslate
noyaca
To use the x-robots-tag, you’ll need to have access to either your website’s header .php, .htaccess, or server access file.
When to use the x-robot-tag:
- Controlling the indexation of content not written in HTML (like flash or video)
- Blocking indexation of a particular element of a page (like an image or video). But not of the entire page itself
- If you don’t have access to a page’s HTML (specifically, to the <head> section). Or if your site uses a global header that cannot be changed
- You do not need to use both meta robots and the x-robots-tag on the same page – doing so would be redundant.
The Robots Meta tag has four main values for search engine crawlers:
- FOLLOW – The search engine crawler will follow all the links in that web page
- INDEX – The search engine crawler will index the whole web page
- NOFOLLOW – The search engine crawler will NOT follow the page and any links in that web page
- NOINDEX – The search engine crawler will NOT index that web page
Other values are:
- NOIMAGEINDEX – Disallow search engines from indexing images on the page
- NOARCHIVE – Prevents the search engines from showing a cached copy of this page in their search results listings
- NOCACHE – Same as noarchive but only used by MSN/Live
- NOSNIPPET – Prevents the search engines from showing a text or video snippet of this page in the search results, and prevents them from showing a cached copy of this page in their search results listings
- NOTRANSLATE – Prevents search engines from showing translations of the page in their search results
- NOCAYA – Prevents the search results snippet from using the page description from the Yandex Directory
HTML Code Robots Example
Not all pages of your WordPress website need to be indexed by the search engines. That is why Yoast gives you the possibility to decide which pages can or can’t be indexed.
The Yoast SEO plugin has a special checkbox, also knows as the “Noindex subpages of archives“. It is easier to add metatags by using this checkbox. Go to Advanced features and select noindex in the empty field right after Meta Robots Index.
Affordable & Effective
We offer affordable SEO services that are fast and effective.
DIY MetaTags
It's not that difficult to create meta-tags. We've made it simple for you by offering you this free meta-tags wizard.
Boost My Metatags
You've created the perfect meta-tags. Now it's time to let the world know your website is open for business.
Create My MetaTags
Rely on the experts. We create the perfect meta-tags for you. Build around the perfect keywords attracting the right customers.
Need some professional
SEO help?
Contact our Customer Service representative to make an appointment and to answer all your questions!
SEO & Online Marketing blog
Extracting Actionable Insights from Data in 2023
How Dynamic Content Improves Website Conversion
