Well-behaved search engines will publish a specific user agent to the web server. I think it'd be much easier to find the official list of search engine user agents than IP addresses.

I suggest you do a google search on 'search engine robot user agents'.

FWIW, this link looks pretty complete and has IP addresses for a few.

I'm also interested in this because we have three different user states:

• Visitor -- No Cookie, Not logged on
• Guest -- Has Cookie, logged on as Guest
• Member -- Has Cookie, logged on paying Member

At this point, all robots are Visitors and we show visitors intro stuff. But much of our content is available only to members and a smaller subset to Guests.

For example, if a member tries to send a visitor a link to member only content in the bboards, the visitor is redirected to a logon page or a page saying that they need to become a member.

So, we want the search engines to index our stuff and I've though of having some robot only pages that can do graceful redirects for real users who use the SE's.

Other thoughts on how to handle this?

Thank you
-Bob We serve Different stuff to users based on

Search Engines cache pages for indexing. They can't legally serve full pages of your content, or have I missed something. Otherwise I'll declare myself a search engine and start serving up NBC news.

Google definitely caches pages for display.  It was very handy when Arsdigita did their Orwellian rewriting of the bulletin board content during their dispute with PhilG.  ALthough they removed content from their site, it was still visible from Google's cache.

Of cource ACS has Robots detection. http://serverspace.com/doc/robot-detection , however I fear that relying on the easy to "fake" USER_AGENT variable could be a problem.

Allan Regenbaum kindly sent me a repair to robot-detection;


repair of robots facility on 3.x

couple changes required to make robots work ...

first .. osme useragents are too long so ..

SQL> alter table  robots modify ( robot_useragent varchar(200));

second, the call to get the file in /tcl/ad-robot-defs.tcl
ad_replicate_web_robots_db  needs to change from

    set result [ns_geturl $web_robots_db_url headers]

    set result [ns_httpget $web_robots_db_url]


The new URL to get a list of robots has changed per the response to Malcolms
post...

In your service.ini


[ns/server/yourservername/acs/robot-detection]
; the URL of the Web Robots DB text file
WebRobotsDB=http://www.robotstxt.org/wc/active/all.txt    < this is the
new URL
; which URLs should ad_robot_filter check (uncomment to turn system on)
 FilterPattern=/ecommerce/*                  >>> will cause a robot check on
any vist to /ecommerce (as an example)
; FilterPattern=/members-only-stuff/*
; the URL where robots should be sent
RedirectURL=/robot-heaven/                   < create this directory with
pages which suit the robots
; How frequently (in days) the robots table
; should be refreshed from the Web Robots DB
RefreshIntervalDays=30

Google offers the cache (on the last line of each listing is a "Cached" link) regardless of whether or not the site is up or has new content.  The direct link to the site is certainly more obvious, and will probably be followed by the naive user over the cached link.  (The cached page includes a note that it may be stale and a link to the current content on your site.)  If you're getting requests for images without their corresponding pages, you might be seeing someone looking at Google's cache of a page (since it will grab images from your site normally).  Or of course someone might have linked to one of your images for use on their website.  Isn't bandwidth theft fun?

I see no sign of any attempt by Google to determine if your site is up before offering the cache.  (Really, would that be feasible?  It wouldn't be fast!)

As soon as you let caching robots skip registration, you are allowing any savvy user to do the same (at least for READING your content) by using Google's caching feature.  Old content which isn't available any more does seem to disappear from Google.  And content which is changed on my site does seem to turn up in changed form on Google in a month or less, as expected.

You're talking about the Internet Wayback Machine, a spin-off of Brewster Kahle's Alexa project: web.archive.org

(FWIW, I may have been the first one to mention it on openacs.org when I described how I found my ACS pages (Problem Set Zero, etc.) after even Google's cached version of them had vanished).

Jerry,

I had no idea that you worked at Google!

It seems then that you are the perfect person to ask about the robot detection module for a difinitive answer.

Originally PhilG described in his book how the robots module would detect a web crawler and serve customised textual content to it instead of the full or password protected page.

Does Google in fact revisit pages anonymously and compare them to pages retrieved by the googlebot to detect masquerading? If so am I correct in assuming that the robot detection package, though still in OpenACS 4.6.3 is no use any more?

Regards
Richard