News Internet Archive announces broader crawler scope

Webmasters never HAD control. Robots.txt was not an enforcable access control mechanism, merely a polite request. It worked only because most search engines agreed to abide by said requests. Apparently, some of the chinese ones use robots.txt exactly backwards, as a map of what parts of the page to crawl FIRST. Which is, you know, the obvious first thing to do once people start acting like a polite request is a real access control mechanism.

Most of the controversy is from people that simply don't understand the difference and think Internet Archive is somehow hacking every server on Earth to bypass the robots.txt firewall. It isn't even setting a bad precedent, since they're far from the first major bot to ignore robots.txt(or use it as a sitemap)

...

Tangentally, have they ever explained why they honor the CURRENT robots.txt file when trying to view previously-stored content? It always seemed to me that they should honor the robots.txt in effect at the time the site was saved, if anything.

 

I would think this is essential functionality? Wouldn't most robots.txt files prevent crawlers from hitting scripts, styles, etc.? Wouldn't they be pretty useful in rendering the pages back at a later date?

 

Ever since the big companies and the politicians found out about the archive and started to force them to remove all kinds of stuff it's really not the same anymore.