FIX message rate monitoring

Question

I am a newby to the site, have limited scripting skills, but able to pick my way through scripts without a problem. I would like to write a script to monitor the FIX messages coming through a number of log files in real time; segregated by account & symbol. The rate needs to be calculated on a per-minute basis. At the moment I am not sure whether this is a minute by minute calculation or a rolling 60 seconds calculation. I haven't written anything yet, I am just looking to see if this is possible and if anyone can give me some pointers as to what would be the best scripting language to employ. Thanks

Answer 1

Here is a brutal solution in gawk. If there is a 35=D on the line we use regexes to split interesting parts out, the timestamp (without the seconds so entries fall into equivalence classes on the minute level), and the two tags and dump it into a 'multidimensional' array, meaning we use these as indices of the array. Once we went through all the messages we scan the array, in no particular order, and dump the counters. It is terribly ugly..the three 'match' functions should be written as one, and perhaps the output sorted, but that's trivial in the shell with 'sort'.

#!/usr/bin/awk -f

#Out_Vec__PWKBVSP-LE2__0 [ 601] : timestamp=2013-08-12-13:00:01.235605858 :: latency=1323.3460000000 :: 8=FIX.4.4|9=0253|35=D|34=0000601|52=20130812-13:00:01.235|49=SENDER|56=RECEIVER|‌​57=SOR|50=TRADER|128=SPSE|11=ORDERID1|453=3|448=16|447=D|452=7|448=DMA1|447=D|452‌​=54|448=ABC|447=D|452=36|1=ACCOUNT123|55=LPSB3|54=1|60=20130812-13:00:00.000|38=6‌​400|40=2|44=17.8700|15=BRL|59=0|10=010| :: aux_len=0,

/35=D/ {
    n=match($0, /.*\|1=([^\|]+)\|.*/, tmp1);
    n=match($0, /.*\|55=([^\|]+)\|.*/, tmp2);
    n=match($0, /[^:]+: timestamp=([[:digit:]]+)-([[:digit:]]+)-([[:digit:]]+)-([[:digit:]]+):([[:digit:]]+).*/, ts);
#    print tmp1[1], tmp2[1], ts[1], ts[2], ts[3], ts[4], ts[5];
    aggr[tmp1[1], tmp2[1], ts[1], ts[2], ts[3], ts[4], ts[5]]++;
}

END {
    for (i in aggr)
    print i, aggr[i];
}

For the samples I get:

ACCOUNT123PSSA3201308121301 3
ACCOUNT123CPFE3201308121301 1
ACCOUNT123LPSB3201308121300 1
ACCOUNT123GETI4201308121301 1

which could be further processed.