Timelion series from cumulative sum of array length

Question

I have documents, that look similar to below:

{
    dateTime: /* My time field */,
    message: {
        users: ['1', '2']
    },
    messageType: 'test'
}

I'd like to construct a timelion series chart that shows me a cumulative sum of the count of the array message.users. My first inkling was to create a script:

if(doc.containsKey('message.users')) {
    return doc['message.users'].length;
} else {
    return 0;
}

From what I could tell, doc.containsKey('message.users') always was false, which tells me that it may not have been indexed correctly. I've tried numerous Timelion, all to no avail:

.es(index=logstash-*,timefield='dateTime',q='messageType:UserList').label('Users Online')

I index my document through the c# NEST api like so:

elasticClient.Index(
    new
    {
        DateTime = DateTime.Now,
        Message = evt.EventArgs.Message,
    },
    idx => idx.Index($"logstash-{evt.MessageCode}"));

Answer 1

I suggest to add another field called userCount to your documents so you don't need to mess with scripting (+ it'll be more performant).

So your documents should look like this:

{
    dateTime: /* My time field */,
    message: {
        users: ['1', '2']
    },
    userCount: 2,                  <--- add this field
    messageType: 'test'
}

Solution 1:

You'd need to change your code a tiny bit to this:

elasticClient.Index(
    new
    {
        DateTime = DateTime.Now,
        Message = evt.EventArgs.Message,
        UserCount = evt.EventArgs.Message.Users.Length
    },
    idx => idx.Index($"logstash-{evt.MessageCode}"));

Solution 2:

If you're using ES 5, you can leverage the Ingest API in order to create a pipeline that will automatically add that userCount field for you. You don't have to change anything in your code.

PUT _ingest/pipeline/user-count-pipeline
{
  "description" : "Creates a new userCount field",
  "processors" : [
    {
      "script": {
        "lang": "painless",
        "inline": "ctx.userCount = ctx.message?.users?.size() ?: 0"
      }
    }
  ]
}

Then, in Timelion, it'll be very easy to chart what you need using metric='sum:userCount' to sum the userCount values and the cusum() function to get the cumulative sum of the userCountover time. The whole expression would look like this:

.es(index=logstash-*,timefield='dateTime',q='messageType:UserList',metric='sum:userCount').label('Users Online').cusum()

Using a few sample documents, the time series looks like this, which seems to be what you're looking for.