aD's design for site-wide search simplifies things in what I feel
is a useful way. They define triggers on each table that they want
to have searched. Any time data is inserted, updated or deleted
from one of those tables, these triggers perform the same action
on the search table. So when it comes time to search, you only
have to do it in one (indexed) table. There are support tables
which supply information such as how to construct links for hits
on data from a particular table.
This makes it fairly trivial to control what gets indexed and what
doesn't. The drawback is duplication of data, of course, but
personally I'm ok with that since the alternative is trying to "teach"
an indexer about the structure of your particular database - Not
Fun.
Conceptually, I think what's needed is to take a sophisticated
search engine like htDig and translate it's search algorithm into
SQL. So instead of generating a regexp or whatever it does now
to turn your query into something to be executed, it would have to
write SQL instead. Then it shouldn't be *too* hard to run that
SQL against the database and return the results. Of course I'm
probably overlooking something horrendously complicated or
this would have been done already! :)
BTW I agree with Dave, searching static pages just isn't good
enough for the kind of sites we are building here. Not to mention
that writing all the dynamic pages to disk represents even more
duplication than the search table method does!
