Topics

Security

Security

All Security Cyber-crime Patches Research CSO

Off-Prem

Off-Prem

All Off-Prem Edge + IoT Channel PaaS + IaaS SaaS

On-Prem

On-Prem

All On-Prem Systems Storage Networks HPC Personal Tech CxO Public Sector

Software

Software

All Software AI + ML Applications Databases DevOps OSes Virtualization

Offbeat

Offbeat

All Offbeat Debates Columnists Science Geek's Guide BOFH Legal Bootnotes Site News About Us

Special Features

Special Features

All Special Features Cloud Infrastructure Week Cybersecurity Month Blackhat and DEF CON Sysadmin Month The Reg in Space Emerging Clean Energy Tech Week Spotlight on RSA Energy Efficient Datacenters

Vendor Voice

Vendor Voice

Vendor Voice

All Vendor Voice Amazon Web Services (AWS) Business Transformation Amazon Web Services (AWS) New Horizon in Cloud Computing DDN Google Cloud Data Transformation Hewlett Packard Enterprise: AI & ML solutions Hewlett Packard Enterprise: Edge-to-Cloud Platform Intel vPro VMware

Resources

Resources

Whitepapers Webinars & Events Newsletters

User topics

Article topics

Reply to post: Checksums?

Robots.txt tells hackers the places you don't want them to look

Tuesday 19th May 2015 07:43 GMT Anonymous Coward

Checksums?

Perhaps robot.txt should include crc32 checksums of directories that should be excluded, so that there is no reference to the actual directory.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

Email

Your email address is never published

Password

Forgotten password ?

Remember me on this computer? Post anonymously?

Enter your comment
Add an icon

Title

Anonymous cowards cannot choose their icon

None

The Register Biting the hand that feeds IT

About Us

Contact us
Advertise with us
Who we are

Our Websites

The Next Platform
DevClass
Blocks and Files

Your Privacy

Cookies Policy
Privacy Policy
Ts & Cs
Do not sell my personal information

Situation Publishing

Copyright. All rights reserved © 1998–2024