How to complete HTTP requests without stopping EventMachine?

412 views Asked by At

I want to use Ruby EventMachine and em-http-request to run parallel HTTP synchronous requests triggered from different threads.

The idea is to run a single reactor in its own thread and push HTTP requests to complete on its event queue thanks to EM.next_tick.

The pattern for every call could be

def complete_request(url, options)
  Thread.new { EM.run } unless EM.reactor_running?
  EM.run do
    EM.next_tick do
      con = EventMachine::HttpRequest.new(url, options[:connection_headers])
      http = com.setup_request(verb, head: options[:headers], body: options[:body])
      http.errback { }
      http.callback { }
    end
  end
  # wait for request completion (but how?)
  ...
end

reqs = []
responses = []
reqs << Thread.new { responses << complete_request('http://www.stackoverflow.com', verb: get) }
reqs << Thread.new { responses << complete_request('http://www.flickr.com', verb: get) }
reqs.each { |req| req.join }    

To make the requests synchronous, I tried to use Fibers but unsuccessfully. Either the request fails to connect or it completes but never exits the event loop.

I don't want to call EM.stop in the callbacks because it would screw up other requests being executed in parallel I guess, and would also stop the reactor while I want it to run until I decide no more requests should be treated.

Does anyone already try to use EventMachine and em-http-request this way ? Can EventMachine support this use case ?

0

There are 0 answers