We're in the midst of reworking the Learning Commons website, and one of the changes is dropping to static files for most of the site (rather than the dynamically generated site we use now). One major thing we change by doing this is the software to search the site.
I've just installed ht://Dig on commons, and it seems to work quite well. I had to compile from source, which I couldn't do on commons itself for some reason (no dev. tools installed on MacOSX Server 10.3?) - I compiled on my TiBook and moved the binaries etc. to commons after testing that it worked.
The package comes with a plain-vanilla test search page, which shows it works quite well. I've had to tell it to not index session-based pages generated by WebObjects applications (ignore urls with "wosid=" in them) so it doesn't wind up with an infinite number of unique and invalid pages in the index.
Anyway, ht://Dig seems to be used quite a bit Out There, and it seems to work quite well In Here. I've set up a crontask to re-index the server every morning at 3am.
