Partial Indexing of an XML file (Bleve)

Question

I am evaluating a couple different libraries to see which one will best fit what I need.

Right now I am looking at Bleve, but I am happy to use any library.

I am looking to index full files except specific ones which are in XML format. For those I only want Bleve to index specific tags as most of the tags are worthless to search. I am trying to evaluate if this is possible but, being new to Bleve, I am not sure what part I need to customize.

The documentation is very good, but I can't seem to find this answer. All I need is an explanation with keywords and steps, no code is required, I just need a push as I have spent hours spinning my wheels with google searches and I am getting no where.

Answer 1

There are probably many ways to approach this. Here's one.

Bleve indexes documents which are collections of key/value metadata pairs.

In your case, a document could be represented by 2 key/value pairs: name of .xml file (to uniquely identify the document) and content of the file.

type Doc struct {
    Name string
    Body string
}

The issue is that body is XML and Bleve doesn't support XML out-of-the-box.

A way to address it would be to pre-process XML file by stripping unwanted tags and content. You can do it using encoding/xml standard library.

For an example of a similar task you can see the code of https://github.com/blevesearch/fosdem-search/

In there they index file in custom format (https://github.com/blevesearch/fosdem-search/blob/master/fosdem.ical) by parsing it into a format they can submit to Bleve for indexing (https://github.com/blevesearch/fosdem-search/blob/master/ical.go).