Philip Greenspun wrote about this in a chapter of his book:
http://www.arsdigita.com/books/panda/publicizing.
He publishes some procedures to overcome what you point at in that chapter and I have changed them to suit PostgreSQL. I suppose the db will be hit quite hard when a robot finds your site though. I took the liberty of placing the file here...
