Google doesn't let you spider their site, but they offer a SOAP-based API for noncommercial use IIRC. I've used it via the Perl module, and it works as advertised.
http://www.google.com/apis/
Also, check out spyonit.com - they have something like what you're looking for, but it's not programmable.