I am trying to implement an Expectation Maximization algorithm for document clustering. I am planning to use Lucene Term Vectors for finding similarity between 2 documents. There are 2 kinds of EM algos using naive Bayes: the multivariate model and the multinomial model. In simple terms, the multinomial model uses the frequencies of different words in the documents which the multivariate model just uses the info of whether a word is present or not in the document(a boolean vector).
I know that the term vectors in Lucene store the terms present in the current document along with their frequencies. This is exactly what is needed for the multinomial model.
But the multivariate model requires the following: A vector which stores the presence or absence of a particular term. Thus all the terms in all the documents must be handled by this vector.
As an example:
doc1 : field CONTENT has the following terms : this is the world of pleasure.
doc2 : field CONTENT has the following terms : this amazing world is full of sarcastic people.
now the vector that I need should be
< this is the world of pleasure amazing full sarcastic people > ( it contains all the words in all the documents )
for doc1 the value of this vector is <1 1 1 1 1 1 0 0 0 0>
for doc2 the vakue of this vector is <1 1 0 1 0 0 1 1 1 1>
Is there any way to generate such a boolean vector in Lucene?
I would first generate the multinomial vectors, and then process them (maybe their textual representation) to get the multivariate vectors.
If the set of documents is not very small, storing full vectors is wasteful. You should have a sparse representation, because every document contains a small subset of the possible terms.
This blog post describes generating feature vectors from Lucene/Solr documents, although I do not think it goes much farther than what you already did.