How to access various timestamps in Azure Iot Edge Pipeline?

Question

I have 2 simple pipelines in Azure Iot Edge. Now, I am interested in accessing the timestamps added to an iot message at each step/node in the pipeline. For example the 'enqueuingTime' at IoTHub and ASA and 'processedTime' from IoTHub and ASA etc. The following are the pipelines:

1. IoT Edge Device (input data source) -> IoTHub -> Custom Endpoint to Azure Storage + Route -> Azure Blob

   So far, I have found that each iot message is appended to a blob file with a property EnqueuedTimeUtc and SystemProperties.enqueuedTime, in the header. Also, both displays same value. Is this the time when the message is received in the IoTHub or the time when it is internally processed in the IoTHub?

   Moreover, if I use a custom endpoint into Azure storage, messages are batched into a single blob file. Is it possible to access the timestamp when a single message leaves the IotHub and also when it is written to the blob file? (i.e. I want to get something like a IoTHub-Processed-Timestamp or a Blob-Insertion-Timestamp of individual messages.)

2. IoT Edge Device (input data source) -> IoTHub -> Azure Stream Analytics -> Azure SQL Database

   In this case, when I am defining the ASA query, the input is the IoTHub endpoint Messaging. In the Select statement, I can access the whole message header and so, I can also access EventEnqueuedUtcTime and EventProcessedUtcTime, which are the timestamps of message enqueue and process completion inside the Stream Analytics job. Further I can access IoTHub.EnqueuedTime, which I assume gives the timestamp when the message is enqueued in the IoTHub. Is there a way to obtain the timestamp when a message enters from ASA and gets inserted into SQL DB? So far, I am using GETDATE() to automatically attach a time stamp during insertion of a record. Is this a good idea?

   Can anybody please let me know if I have a correct understanding of Azure Iot timestamps? Are there any documentations on how to access all such timestamps?

Answer 1

EventEnqueuedUtcTime is the default timestamp of events coming from an IoT Hub in Stream Analytics is the timestamp that the event arrived in the IoT Hub. EventProcessedUtcTime is the date and time that the event was processed by Stream Analytics. More info this document. As you mentioned when you use stream data from an IoT Hub, you have access to the EventProcessedUtcTime and EventEnqueuedUtcTime in your Stream Analytics query.

When a single data stream contains multiple event types having timestamps in different fields, you can now use TIMESTAMP BY with expressions to specify different timestamp fields for each case.ASA is a temporal system, so every event that flows through it has a timestamp. A timestamp is assigned automatically based on the event's arrival time to the input source but you can also access a timestamp in your event payload explicitly using TIMESTAMP BY.

Update: In addition, EventEnqueuedUtcTime is the date and time that the event was received by the IoT Hub. But IoTHub.EnqueuedTime is the time when the message was received by the IoT Hub.IoTHub.EnqueuedTime is not supported on the telemetry path when a Device/DeviceClient sends a telemetry message to IoTHub.Azure IoT Hub provides the capability to stream data from your connected devices, and integrate that data into your business applications. IoT Hub offers message routing and event routing for integrating IoT events into other Azure services or business applications.More info here.