Forum OpenACS Q&A: Response to Search Engine cloaking / IP delivery

Posted by John Sequeira on
Well-behaved search engines will publish a specific user agent to the web server. I think it'd be much easier to find the official list of search engine user agents than IP addresses.

I suggest you do a google search on 'search engine robot user agents'.

FWIW, this link looks pretty complete and has IP addresses for a few.