robot.txt it is important?

siva_aarya · Joined: 11 Apr 2007 Posts: 4 Location: Chennai

In all of my websites, there is no robots.txt. But tag <META NAME="ROBOTS" CONTENT="INDEX,FOLLOW"> and <META NAME="ROBOTS" CONTENT="all"> both are used. Is it enough or I need to add robots.txt. How important is this file. How search engine treats / uses it. If it is necessary to add Robots.txt what should be the code included in this file?

Google is visiting only home page of my website and goes back without following links on that page. Is this is because of robots.txt. If no what can be the reason. CONFUSED

Thank you

http://www.g-antssoft.com/

Gloria · Guest

You can learn basic things about robots.txt from wikipedia

http://en.wikipedia.org/wiki/Robots.txt

abbysmith11 · Joined: 21 Jul 2011 Posts: 20 Location: Nederland

A Robots.txt file is helpful to keep out unwanted search engine spiders like email retrievers, image strippers, etc. It defines which paths are off limits for spiders to visit. This is useful if you want to hide some personal information or some secret files.

gratis casino bonus

KellyWatson · Joined: 25 Jul 2011 Posts: 2

Yes, robots.txt is important for SEO, With the help of robots.txt we can block those pages, which we don't want o cached by search engines.

lizatailor23 · Guest

Yes, Robots.txt is important. It's a text file in which it gives information to the google crawls that which pages not to visits or not crawled.

yourmobisite · Joined: 11 Jul 2011 Posts: 6

Hello..

When a search engine crawler comes to your site, it will look for a special file on your site. That file is called robots.txt and it tells the search engine spider, which Web pages of your site should be indexed and which Web pages should be ignored.

nancy1 · Joined: 27 Jul 2011 Posts: 14

One way to tell search engines which files and folders on your Web site to avoid is with the use of the Robots metatag.

stevejustine · Joined: 06 Jan 2012 Posts: 18 Location: Ahmedabad

I dont think robot.txt is absolutely necessary, but its an addon if you have it..Its really simple to put up one. Smile

web design | SEO Company India