I am a new Elasticsearch user, but I am struggling to accomplish something that was easy for me in Splunk. There are a few specific fields that I want from each event in my search, but the search "hit" outputs are always returned in a big json structure that is 95% useless for me. I do my searches with the python requests module, so I can parse the results I want in python when they return, but I have to access millions of events and performance is important, so I hope there is a faster way.

Here is an example of one single event returned from an Elasticsearch search:

<Response [200]>                                                                             
{                                                                                            
    "hits": {                                                                                
        "hits": [                                                                            
            {                                                                                
                "sort": [
                    1559438581000
                ],      
                "_type": "_doc", 
                "_source": {                                                                 
                    "datapoint": {                                                           
                        "updated_at": "2019-06-02T00:01:02Z",                                
                        "value": 102
                    },  
                    "metadata": {                                                            
                        "id": "AB33",                                            
                        "property_name": "some_property",              
                        "oem_model": "some_model"                                       
                    }
                },  
                "_score": null, 
                "_index": "datapoint-2019.06",                                               
                "_id": "datapoint+4+314372003"                                               
            }, 

What I would prefer is for my search to return only results in a table/.csv/dataframe format of the updated_at,value,id,property_name,oem_model values like this:

2019-06-02T00:01:02Z,102,AB33,some_property,some_model
..... and similar for other events ...

Does anyone know if this is possible to do with Elasticsearch or with the requests library without parsing the json after the search output is returned? Thank you very much for any help.

0 Answers