PYTHON: How to receive data continuously from a server that's using zmq Poller?

Question

I am using simple requests.post() module to connect to a server and receive data

ack = requests.post('<ip>/get_data, data=data, timeout=10.0, verify=False)

Below is a method get_data() in server that's using zmq poller to receive data until final response is obtained.

def get_data():
    req = json.loads(self.params.get('data'))
    wire = wiring.Wire("indexing_data_pool", zmq_context=g.zmq_context)
    try:
        close_immediately = True
        poll_agent = zmq.Poller()
        poll_agent.register(wire.socket, zmq.POLLIN)
        wire.send(req)
        iteration = 1
        while True:
            socks = dict(poll_agent.poll())
            if wire.socket in socks and socks[wire.socket] == zmq.POLLIN:
                res = gevent.with_timeout(10, wire.recv, timeout_value=None)
                if res.get('final'):
                    log.warn('Last Iteration: %s, Length of Rows: %s' %(iteration, len(res['rows'])))
                    break
                else:
                    log.warn('Iteration: %s, Length of Rows: %s' %(iteration, len(res['rows'])))
                    next_req = {'search_id': req['search_id'], 'seen_version':res.get('response_version')}
                    wire.send(next_req)
                iteration +=1
    finally:
        wire.close(immediate=close_immediately)
        poll_agent.unregister(wire.socket)

In server side, the response obtained is logged as below:

2017-08-28_07:17:55.43370 WARNING: Iteration: 1, Length of Rows: 100
2017-08-28_07:17:55.44269 WARNING: Iteration: 2, Length of Rows: 100
2017-08-28_07:17:55.44894 WARNING: Iteration: 3, Length of Rows: 100
2017-08-28_07:17:55.45742 WARNING: Iteration: 4, Length of Rows: 100
2017-08-28_07:17:55.46327 WARNING: Iteration: 5, Length of Rows: 100
2017-08-28_07:17:55.46687 WARNING: Iteration: 6, Length of Rows: 100
2017-08-28_07:17:55.47074 WARNING: Iteration: 7, Length of Rows: 100
2017-08-28_07:17:55.47658 WARNING: Iteration: 8, Length of Rows: 100
2017-08-28_07:17:55.48385 WARNING: Last Iteration: 9, Length of Rows: 75

So, I assume that the zmq poller I have implemented in server side is working perfectly fine. But, I am curious to know how to send back these 9 iterations of data back to the requesting client?

P.S. I want to receive the data continuously in client side. You may suggest appending each batch of response somewhere and sending the final response back to the client. This won't be feasible when the response is too big (The requesting client would get timeout)

Answer 1

Ok, I handled this using Response object from Flask module. Invoking a separate method that'd yield the data continuously and making the final Response object and returning it solved my problem.