I'm using ht://dig to do something like this. (
www.htdig.org) Basically, the
ht://dig configuration file lets you specify a "start" page and then
the depth of links to follow. So what I did was I created a Tcl page
that links to all of the useful content on the site and set the link
depth to one. So far it works pretty well.