Know about The Importance of robots.txt

Importance of robots.txt

What is Robots.txt :

Robots.txt is a simple text file provided essential information for Search Engines to Navigate and Index a website & improve search engine rankings for the site.

How it works?

When a search engine’s crawls our site, search engine spider look for this particular file. This file tells the search engine spider, to priorities the web pages to be indexed. In the sense, what pages of our site should be indexed and what pages should be discarded.

Where to Place it?

website’s root directory. : http://www.seo24marketing.com/robots.txt

Examples :

Use the following content for your robots.txt file if you want to allow all search engine spiders to index all files of your Web site:

User-agent: *
Disallow:

Use the following content if you don’t want to index directory

User-agent: *

Disallow: /cgi-bin/

Disallow: /images/

Post Author : Someshwar

Source : Importance of robots.txt

Posted in: On Page SEO