OCaml - Parmap executing Lwt threads hangs on the execution

Question

This is a follow up to this question: How to synchronously execute an Lwt thread

I am trying to run the following piece of code:

open Lwt
open Cohttp_lwt_unix

let server_content2 x  =
    "in server content x" |> print_endline ;
    Client.get (Uri.of_string ("http://localhost:8080/"^x)) >>= fun (_, body) ->
        (Cohttp_lwt.Body.to_string body) >|= fun sc -> sc
        ;;
let reyolo () =
    List.init 10 (fun i -> server_content2 (string_of_int i) ) ;;
let par () = 
    let yolo = reyolo () in
    "in par" |> print_endline;
    Parmap.pariter 
        ~ncores:4 
        (fun p ->  "before run" |> print_endline ; "content:"^(Lwt_main.run p) |> print_endline  ; "after run" |> print_endline ) 
        (Parmap.L yolo);;

par ()

I expected this to perform 10 remote connections. What I get is in par function Lwt_main.run seems to stuck before doing an actual remote call.

I doubt it might be of any significance but the server that suppose to respond is made in python and looks like this:

import subprocess
from bottle import run, post, request, response, get, route

@route('/<path>',method = 'GET')
def process(path):
    print(path)
    return "yolo"

run(host='localhost', port=8080, debug=True)

Answer 1

The issue is that the calls to server_content2, which start the requests, occur in the parent process. The code then tries to finish them in the child processes spawned by Parmap. Lwt breaks here: it cannot, in general, keep track of I/Os across a fork.

If you store either thunks or arguments in the list yolo, and delay the calls to server_content2 so that they are done in the child processes, the requests should work. To do that, make sure the calls happen in the callback of Parmap.pariter.