How can I execute a loop multiple times simultaneously to speed things up?

Question

I have a Python script that does requests to a server, and checks their response. About 10% of the responses are special, so it prints a message when it encounters one. It does 90000 iterations, and I managed to print the current progress in command prompt like this:

print('{0}/{1}{2}{3}{4}{5}'.format(str(current_iteration_number),"90000  ",speed, " requests/s, ready in: ", timeleft, " minutes."),end="\r")

It manages to do about 2.5 requests per second, but I know this could be at least 5 times faster. I tried this by executing the same script 5 times simultaneously (any more would result in the server blocking me for doing what could look like a Ddos attack). Although this works, having to run 5 command prompts at once and manually joining the results is not a nice way of doing things.

How can I let Python execute the 5 loops simultaneously by itself, and print the joined progress and results?

Answer 1

import multiprocessing
import numpy as np
def my_loop(x):
    #some stuff (this here is a silly example since it could be vectorized)
    return x**2
pool = multiprocessing.Pool()
pool_list = [float(i) for i in range(90000)]
results = pool.map(my_loop,pool_list)

Answer 2

You can use threading or multiprocessing library. Besides if your operation is I/O bound you can use asyncio library, which will give you advantages of concurrency in single-thread.