Forum OpenACS Development: Re: Blocking access to login based on repetitive hits by ip address?

Some robots just don't respect robots.txt...

On my personal site, which has had 1386 hits inside /register/ since last October, 924 are from msnbot, 154 from FAST-WebCrawler, 126 from "http://www.almaden.ibm.com/cs/crawler", and another 21 from miscelaneous robot-like UAs... and /register/ is in the robots.txt... at least google seems to be doing the right thing...

jeez... you'd think at least MSN would be capable of respecting standards on the web... :)