In past I used Lucene full-text search engine library for indexing my websites that were build upon Microsoft Content Management Server.
Now that I use Sitefinity I rewrote this search engine library to index the Sitefinity websites. You can find the library at http://sourceforge.net/project/showfiles.php?group_id=119263
I don't know what the status of this library at the moment is. In past I used the specific library for indexing Microsoft Content Management Server. I don't know for sure if this is identical with the current version.