Forum OpenACS Q&A: ACS/PG+AOLserver3 and robot detection...

Has anyone gotten this to work?  I get an error from nsd.geturl,
trying to connect(), it says:

connect() failed: resource temporarily unavailable

when it tries to hit the web page at info.webcrawler.com.  I checked
at their server's up, that's not the problem.

Presumably some silly configuration problem, anyone want to give this
lazy-assed dude a hint?

Collapse
Posted by Michael Feldstein on
Just curious; did you ever figure this one out?
Collapse
Posted by Don Baccus on
It was a silly configuration problem.  I forget what, exactly.
Collapse
Posted by Stan Kaufman on
Any clue what the silly configuration problem was?

The robot updating mechanism is failing with OpenACS 3.2.5, and I don't see any bugs (open or closed) related to this problem. Used to work ok with 3.2.4.

The call to ns_geturl in ad_replicate_web_robots_db seems to fail; it returns 'file does not exist'.

I see that the url for the robot data has changed (now it's http://www.robotstxt.org/wc/active/all.txt) but the file format doesn't seem different, so I presume that ad_replicate_web_robots_db should parse the file correctly -- if it could get it. I've set this new url in the WebRobotsDB parameter in the my server config file.

Any clues about what is going on here? TIA!