Merge Documents based on field value?

Question

I have multiple Documents within an Index, each have the following fields:

id serviceName Type

Now, stupidly, id is not unique and I want to change that. I want to use Kibana/Elasticsearch to query the data so that I have id unique and the behaviour I want is that if I have the following Docs:

id    serviceName    Type
1     A              T1
1     B              T2
1     D              T2

I use a query so that I get this result

1   A,B,C            T1,T2,T3

Is there a way for this?

Answer 1

You cannot do this with just Elasticsearch/Kibana, you have to write some code. You can use the scroll api to iterate through all the documents in the index, and then use an upsert query to index them into a new index. I think your upsert request will look something like this:

POST test/type1/1/_update
{
    "script" : {
        "inline": "ctx._source.serviceName.add(params.serviceName); ctx._source.Type.add(params.Type)",
        "lang": "painless",
        "params" : {
            "serviceName" : "A",
            "Type": "T1"
        }
    },
    "upsert" : {
        "serviceName": ["A"],
        "Type": ["T1"]
    }
}

This means in case id 1 doesn't exist yet, add it with the "upsert" value for the document, otherwise do the script (which appends the serviceName and Type values to the existing doc).

This would be pretty straightforward to do with very little code using elasticsearch-py, check out the scan helper and bulk helper