Data Modeling Issue

Question

just started using Keen.IO…very nice product. I have a question please on data modeling:

I am tracking mobile app registration events that have the following attributes:

Device Id,
Platform, Manufacturer, Marketing Version,
Language and
Latitude and Longitude

The problem is that the mobile app sends this registration every time the device changes location. In a way, it is to maintain the device latest location and keep track of historic changes. I created a new collection for devices and it seems to be OK as I am able to select_unique the keen.location.coordinates to draw the devices on a map.

The question though is how would I know the latest location update? So now I have several events in the collection for the same device (uniquely identified by a device id) but no way of finding the latest!

Thank you for any pointer.

Regards, Khaled

Answer 1

It is technically possible to use existing Keen query types to retrieve the latest position of a device as you require, the problem is doing so is inefficient and doesn't scale well (mainly because Keen is designed to store/query event data, not entity data). You'd have to plug the Device Id you retrieve from the select_unique into individual queries for each device to find the latest location. Doing so would bump you up against concurrency and/or rate limiting fairly quickly.

That being said, that leaves you with a few options:

1. Use Keen's S3 integration. This feature writes all of your raw events to S3 for consumption however you see fit. You could use this data to create a pipeline to a separate entity database which would allow you to retrieve the device's latest location. Note: there is an extra charge for this feature.

2. Post device/location data to a separate entity database at the same time you're sending the event to Keen, and use that database as the source of the device's latest location.

Either of these options will work, it just depends on how you'd like to implement/manage data collection in your application.

Answer 2

I think you can achieve this by combining a select_unique query with a group_by clause and looking at the last item in the result.

Though, as terrhorn noted – it's not a scalable solution. I wouldn't use it for anything other than dashboarding.

Here's a quick example:

var query = new Keen.Query("select_unique", { eventCollection: "devices", targetProperty: "location", group_by: "device_id" });

The result will look something like this:

{ "result": [ { "device_id": "4252f729-7bdc-a487-be15-984999a96683", "result": [ "location_1", "location_2" ] } ] }

NOTE: This is theoretical – I'm not sure if Keen sorts the result of the group_by by the order in which the events came in or not – but it's a good assumption to test. I don't see why they wouldn't.