The project i am working on reads data from a single WebSocket at an incredibly fast rate. The problem i have is the websocket provider requires you to handle the data as quickly as possible to avoid overflowing the temporary buffer at their end.
The issue i have is im using PHP to try and index the documents into elasticsearch on every message we receive from the websocket and this is not quick enough which ends in our socket being closed due to the buffer/queue becoming full on their end.
I'm hoping to seek some help to establish the best possible solution without losing any of the data from the socket.
My theory is:
- Connect to sock and maintain a connection
- Every message received handoff to an async process or logstash that can handle the ingest into elasticsearch without the need for the websocket to wait before sending/handling the next message.
Using elasticsearch PHP to index each message simply won't cut it and after 10 minutes the socket is closed.
Server resources are fine, 16 core, 128gb ram and 10gbps connection. If i remove the elasticsearch index function from the PHP code the socket stays connected fine and server resource usage is around 10%.
Can someone with more experience propose a better solution or potentially have logstash handle this on different threads.
Thanks