Communication between node.js app and worker on IronWorker. How to properly poll a task status and receive task response?

Question

The application is as follows:

First of all, I`m using a node.js app with the express.js framework as web server. On the front-end, when the user sends a payment via a POST form, the express.js routes this task via 'app.post' and calls an external API for payment. This API request should ideally return the URL for which the user should be redirected for payment. Then, the express.js should redirect the user via the 'app.post' response variable (res):

res.redirect(307, url);

Since this payment API request takes some time and since I am using Heroku, the API request times out and an error is returned. To solve this, I know I would need to set a worker on an worker Heroku add-on (according to https://stackoverflow.com/a/11438381/2859410), such as IronWorker. However, I am having trouble to understand what would be the proper mechanism to communicate the URL returned by the external API request made in the worker back to the node.js app.

To make things clearer: - The user submits the payment form.
- The node.js app processes the POST data and calls a worker ("enqueues" a task on a worker).
- The worker requests an external payment API, which returns an URL.
- The worker should communicate this URL back to the node.js app.
- The node.js app should receive this URL and redirect the user.
Yet, these operations have to be non-blocking.

I can think of 2 alternatives for the communication:
Alternative 1 - When the front-end user requests a payment via POST, the node.js app calls the specific worker (also via POST), with the payment data as the payload. The worker process the data, requests the payment API and gets the payment URL. Meanwhile, the node.js performs a polling of the worker status and, when receives the "completed" status, redirects the user.
Fallacy: The difficulty here for me is to perform constant polling in a asynchronous manner. Since I need to redirect the user to the payment URL, I would need to have the URL inside the "app.post('/form', function(req,res) {" method, such that I can the response ("res") to redirect the user. For this, I took a look at async.whilst(test, fn, callback), but I don`t know how could I use a test function that is synchronous. My immediate idea would be a test function that issues a status request and a callback function that redirects the user. However, this test function is an asynchronous operation (for example using 'request' module).

Alternative 2 - The worker also receives the payment data as payload, process the data, requests the payment API and gets the payment URL. When it finishes, it communicates the URL back to the node.js app via an HTTP request.
Fallacy: How would I tie the URL communicated back to the node.js app to the user that requested the payment? In my understanding, the HTTP request with the payment URL would have an independent route than the form POST route. In another words, this way I would not be able to have the payment URL inside the "app.post('/orderform', function(req,res) {" method, and, therefore, would not be able to use response variable ("res") to redirect the user.

Tha node.js app would be structured as follows:

app.post('/order', function(req, res) {  
    // user data on req.body
    // process data and request payment API  
    // Wait until URL is communicated to the endpoint belo '/url'
    // receive URL from the other endpoint and redirect user
}

app.post('/url', function(req, res) {  
    // receive payment URL on req.body
    // communicate the URL to the '/order/ endpoint    
}

What would be recommended in this case?

Thanks in advance.

Answer 1

I'd suggest to use something in between these two alternatives.

1. You receive request from user and queue worker to process it.
2. You start polling some internal storage (e.g. database) for the results of request.
3. When worker need to store result, it calls back your app (different endpoint) which stores data in the local storage (that place where polling operation looks for it).
4. When data is available, you'll be able to redirect users.

Answer 2

I would suggest using a service like pubnub for this (http://www.pubnub.com/) . When you get a request from the user add to the queue to process it and create and return a token to identify this work request. Send that token back to the client. Use that token as the pubnub channel to listen on. When the worker finishes, send a message to that same token to let the client know that the work is done.