Google had been providing a robots.txt validator for a long time. Its a very effective tool since the verification is done right from google's month itself. What ever gets verified in Robots.txt validator is what Googlebot sees. That made robots.txt tool a very attractive tool and almost every webmaster who has validated their site in webmaster central has one point or the other used robots.txt validator to validate their website robots.txt.
I am sure everyone knows what robots.txt is its mostly placed to stop bad robots from entering a website , stealing your email IDs , scraping your website etc
Now google has a robots.txt generator where we can generate a valid Google accepted robots.txt file.
0 Comments:
Post a Comment
Links to this post:
Create a Link
<< SEO Blog Home