Merlin two tower model data training error (ValueError: high is out of bounds for int32)

Question

model.compile(optimizer="adam", run_eagerly=False, metrics=[mm.RecallAt(10), mm.NDCGAt(10)])
model.fit(train, validation_data=valid, batch_size=4096, epochs=3)

ValueError                                Traceback (most recent call last)

C:\ProgramData\Anaconda3\lib\site-packages\merlin\models\loader\backend.py in _shuffle_indices(self)
    326     @annotate("_shuffle_indices", color="darkgreen", domain="nvt_python")
    327     def _shuffle_indices(self):
--> 328         generate_local_seed(self.global_rank, self.global_size)
    329         if self.seed_fn:
    330             new_seed = self.seed_fn()

C:\ProgramData\Anaconda3\lib\site-packages\merlin\core\dispatch.py in generate_local_seed(global_rank, global_size)
    647         cp.random.seed(seeds[global_rank].get())
    648     else:
--> 649         seeds = random_state.randint(0, 2**32, size=global_size)
    650     return seeds[global_rank]
    651 

mtrand.pyx in numpy.random.mtrand.RandomState.randint()

_bounded_integers.pyx in numpy.random._bounded_integers._rand_int32()

ValueError: high is out of bounds for int32

I was trying to train the model usng Nvidia Merlin

Expecting a model iteration step.

Answer 1

Hello here the solution which worked for me:

Go to your folder AppData\Roaming\Python\Python_version_here\site-packages\merlin\core

In the file dispatch.py change the function generate_local_seed as:

def generate_local_seed(global_rank, global_size):
    random_state = get_random_state()
    if cp:
        seeds = random_state.tomaxint(size=global_size)
        cp.random.seed(seeds[global_rank].get())
    else:
        seeds = random_state.randint(0, 2**31-1, size=global_size)
    return seeds[global_rank]

Here we declare the max value now as 2**31-1