Forum OpenACS Q&A: Response to OT: How to search the web...
3: Response to OT: How to search the web... (response to 1)
Posted by Michael A. Cleverly on 09/04/02 08:02 AM
Not that I'd advocate your breaking Google's rules but if you do write an HTTP robot to run queries against Google (or possibly sites that think if you aren't running the latest MSIE you're unfit for their site), just make sure that the User-Agent string that you send isn't "AOLserver" or "Tcl HTTP 2.3" or "Perl::LWP" (or whatever). Stick to something mainstream (IE, Netscape, Mozilla, or Opera, etc.) and you'll be reasonably safe (in general) from sites that discriminate based on the User-Agent header.