Nodejs response to API request after data processing is done

Question

In my current project, my nodeJs/express will receive a HTTP request through a route.

Once received, node will then use NightmareJS to perform webscraping and subsequently execute a python script that further process the data.

Lastly, it would then update this data into MongoDB.

Everything takes about 5mins.

What I am trying to achieve is to allow my front-end to somehow receive an acknowledgement that the request was put through. But also receive an update when the above process is completed and the database is updated.

I have looked into using Long polling or socket.io. However, I don't know which one I should use or how. Or should I use rabbitMQ instead? Putting the response that it is complete into the queue while my front-end constantly querying this queue.

Answer 1

1. Long polling or socket.io are similar, socket.io has Long polling fallback if WS not supported

2. rabbitMQ is quite different, you cannot use rabbitMQ protocal in browser, so you need a client app, not a web

3. socket.io is excellent and go well along express, still there are other options, SSE (server send events), firebase. You need to feel them before you choose one, they are not that hard if you follow their official guide

4.some of my opensource might help

https://github.com/postor/sse-notify-suite

https://github.com/postor/node-realtime-db

5. benefits of each solution

   • ajax + server cache: simple
   • long pull: low latency
   • SSE: low latency, event based
   • socket.io: low latency, event based, high throug put, double direction, long pull fall back