I've been working for a while with ELK and I'm having some trouble tweaking my mind to a non-relational space.
I understand perfectly the advantages about recording separate records of same element for each change in our data source.
But I'm having trouble understanding how to filter all of those records based on the value of a field which is not present in every one of them.
Let me explain this with an example.
We have an index with hundreds of thousands books recording the evolution of notifications to people (we are a public service). So each time we send a notification to anyone we are keeping track of the creation date, sending date, reception date, etc...
This way, we have different books of the same notification in our index:
- A first book when it's created, at this point sending and reception dates are null.
- When it's sent a new book is created, the creation date is the same, but status and reception dates are changed, also, reception date is still null.
- When it's received, a new book is created, with a new status value and finally a reception date
Well, this allows us to make very finely detailed statistics about sending and receiving times, costs, etc...
But, a notification can be cancelled at any point before being received, which means that in that point we create a new book on the same notification where "cancelled" field is true, every previous book on the same notification will have that field as false, and every future book will have it as true.
Now, when I'm creating dashboards, there are lots of them where I'll want to ignore cancelled notifications, but if I filter by cancelled:true, I'll filter only those books created after its cancellation, not those previous, as it's creation book.
So, although I want to ignore cancelled notifications, I won't be able to ignore it, as every one of them will have at least one book where cancelled:false.
If I want to, for example, count the number of cancelled notifications, I just need to count unique notifications where cancelled:true, so each one where there's at least one book with cancelled:true will be counted up, but this doesn't add up on the negative, as I cannot obtain a count of not cancelled notifications.
I'm sure this type of filtering must have a best practice defined, but I've been unable to find anything on it after several days googling.
How can I discard any book of an element where at least one book fulfills a search condition?