When I write some code to test the EventHub which is a newly released on azure service bus.
As there is very few article online and msdn also do not have rich documentation about the detail of event hub. So I hope someone could share your experience for my question.
For EventHub, we have following statement:
we use "offset" to remember where we are when reading the event data from some partition
the event data on the EventHub would be expired (automatically?) after some configurable time span
So my question is, can the offset still be available/durable when some of the event data is deleted as the result of expiration?
For example, we have following data on one of partition:
M1 | M2 | M3 | M4 ( oldest --> latest )
After my processing logic runs, let's say that I have processed M1 and M2, so the offset would be the start of M2(when use exclusive mode).
After some time, and if my service is down during that time. M1 is deleted as the result of expiration. so the partition would become:
M2 | M3 | M4 | M.... ( oldest -> latest )
In this case, when my server is restart again, is the offset i stored before is still be available to be used to read from M3?
We can also image this case on runtime when my consumer server is reading the event data on eventhub when some of the oldest event data is expired, does the offset still be available on runtime?
Thanks for any sharing of this question.
Based upon how various parts of the documentation is written I believe you will start from the beginning of the current stream as desired if your starting offset is no longer available. EventProcessorHost should follow similar restrictions. Since the sequence numbers are 64 bits, I would expect one of those to be able to serve as an offset within a partition since they monotonically increase without being recycled. The offset should have a similar property. So if EventHubs are designed in a reasonable fashion (ie like similar solutions), then the offsets within partitions can hold despite data expiration. But since I have not yet tested this myself, I will be very unhappy if it is not so, and I'd expect an Azure person to be able to give true confirmation.