I need to implement feeds arbitration for FAST protocol. Problem is pretty common and there are even hardware solutions As problem is wide known i do think there should be at least general suggestions of how to implement that (how many queries should I use, how many ring-buffers, how many readers, when to drop packets etc.) and probably someone can point me to some implementation. For those who not familar with FAST I add some description:
Data in all UDP Feeds are disseminated in two identical feeds (A and B) on two different multicast IPs. It is strongly recommended that client receive and process both feeds because of possible UDP packet loss. Processing two identical feeds allows one to statistically decrease the probability of packet loss. It is not specified in what particular feed (A or B) the message appears for the first time. To arbitrate these feeds one should use the message sequence number found in Preamble or in tag 34-MsgSeqNum. Utilization of the Preamble allows one to determine message sequence number without decoding of FAST message. Processing messages from feeds A and B should be performed using the following algorithm:
- Listen feeds A and B
- Process messages according to their sequence numbers.
Ignore a message if one with the same sequence number was already processed before.
// tcp recover algorithm further
So I think that solution should be like that:
For each of two feed create dedicated thread and dedicated buffer. Add data to buffer as data arrives. (should it be ring-buffer or queue or what?)
Create "reader" which "spin" and checks both thread for the last available "sequence number". As soon as "sequence number" is available next packet need to be processed and both threads should drop it after that.
Any suggestions of how to implement algorithm itself and probably suggestion which structures to use are welcome. In particular probably someone can suggest lock-free queue / ring-buffer implementation.