Sign in / up

Reply to post: Re: Yes.

Robots.txt tells hackers the places you don't want them to look

Daggerchild Silver badge
Reply Icon
Flame

Re: Yes.

I always assumed robots.txt was just to flag parts of the tree that were not worth the robot's trouble
Or "Please don't spider this area as the backend app/database is so bad it will cripple the whole site"

POST COMMENT House rules

Not a member of The Register? Create a new account here.

Forgotten password ?

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon