I'm trying to get numba, multiprocessor and random number generators work together. I have downsized my real problem to the following piece of code containing the important elements. The following works for me.
import numpy as np
from numba import jit
import multiprocessing as mp
#@jit(nopython=True)
def compute_with_random(j,rng):
x=rng.normal(0,0.3,j)
y=np.sum(x)/j
return y
#@jit(nopython=True)
def single(args):
(n,se)=args
rng = np.random.default_rng(se)
s=0
for i in range(1,n):
s+=compute_with_random(i,rng)
return s
def Call_Multi():
seed_sequence = np.random.SeedSequence(12345)
seeds = seed_sequence.spawn(4)
all_ins=[ (500,seeds[0]), (700,seeds[1]), (400,seeds[2]), (200,seeds[3]) ]
pool = mp.Pool(4)
result = pool.map( single, all_ins )
return result
if __name__=='__main__':
print( Call_Multi() )
As for my real problem the two functions compute_with_random() and single() take quite long and numba can accelerate them I want to use the numba decorator, so using the @jit decorators above result in the following error.
numba.core.errors.TypingError: [1mFailed in nopython mode pipeline (step: nopython frontend)
[1m[1mnon-precise type pyobject[0m
[0m[1mDuring: typing of argument at C:\test\test_rng_numba.py (16) [0m [1m
File "test_rng_numba.py", line 16:[0m
[1mdef single(args):
[1m (n,se)=args
[0m [1m^[0m[0m
This error may have been caused by the following argument(s):
... (truncated)
If I replace x=rng.normal(0,0.3,j) by x=np.random.normal(0,0.3,j) and remove the arguments rng in compute_with_random() and se in single() the example works also fine with numba. So probably there is a problem with numba and the parallel random number generator, or the seeds/rng. The random number generator with rng.normal() is suppossed to make independent chains of random numbers for each process. Any ideas how to solve that issue? thanks