ElasticSearch: shingles - match phrase if field contains exact shingle token

Question

I am newbie with elastic search and have trouble with following scenario:

1. Let's consider I have 2 documents which contains only one field "text"
   1. "text" : "token1 token4"
   2. "text" : "token2 token3"
   3. "text" : "token4 token5"
2. And by following query-text "token1 token2 token3 token4 token5" I want to find only documents 2 and 3

I need something similar to shingles filter which will create following tokens from query:

["token1 token2", "token2 token3", "token3 token4", "token4 token5"]

And will make exact match by these tokens, so tokens "token2 token3" and "token4 token5" will match document

Thanks in advance!

Answer 1

This can be done by using shingle filter and making output_unigrams false(It will prevent generation of single token). Create your index like this

PUT shingle_index
{
  "settings": {
    "analysis": {
      "analyzer": {
        "shingle_analyzer": {
          "tokenizer": "standard",
          "filter": [
            "shingle_filter",
            "lowercase"
          ]
        }
      },
      "filter": {
        "shingle_filter":{
          "type" : "shingle",
          "max_shingle_size" : 2,
          "min_shingle_size" : 2,
          "output_unigrams" : false,
          "output_unigrams_if_no_shingles" : true
        }
      }
    }
  },
  "mappings": {
    "mytype":{
      "properties": {
        "text" : {
          "type": "string",
          "analyzer": "shingle_analyzer"
        }
      }
    }
  }
}

Index some sample documents.

POST /shingle_index/mytype/_bulk
{"index":{"_id":5}}
{"text":"token1 token4"}
{"index":{"_id":3}}
{"text":"token2 token3"}
{"index":{"_id":2}}
{"text":"token4 token5"}

Then simple match query will give you the desired result.

GET shingle_index/_search
{
  "query": {
    "match": {
      "text": "token1 token2 token3 token4"
    }
  }
}