Spring XD or Integration to parse log stats in real time

Question

I have a dashboard web application that currently starts a thread and tails a log file, even time a line is added to the file the tailer picks it up, parses it and then publishes an event around the application, which in turn does things like send it to the client over a web socket or updates a total stored on disk.

This all works fine and so far seems to be handling 500,000 log events a day without batting an eyelid.

But looking at the Spring family maybe there is a better way of doing this in a more 'standardised' fashion that can make it easier to maintain the code and for others to support it.

As I'm new to Spring can someone tell me if this sort of thing is best suited to Spring Integration or Spring XD, or should I take it a stage further and use Spring Integration and Rabbit MQ?

Answer 1

Spring Integration 3.0 now has a tail inbound adapter. It will be released soon, the release candidate was announced last week. XD uses it in its tail source module.

Answer 2

I'am doing quite the same job with Spring XD. processing 30Go of logs files a day. Enrich them, send them to hadoop and Elasticsearch.

We are really happy with that technology that mixes spring integration, spring batch in a distributed way.

I've noticed that Redis was a real bottelneck and switched to RabbitMq to have a better throughtput