Why am I getting NaN returned from the categorical_crossentropy which causes an index error when using pyGAD for optimizing a keras ANN parameters?

Question

I've been trying to implement a GA to optimize the parameters of my ANN. I'm new to both of these libraries and I've used this as help to implement it https://blog.paperspace.com/train-keras-models-using-genetic-algorithm-with-pygad/. I'm using: pygad version 3.0.1 for the GA, Tensorflow keras version 2.12.0 for the ANN model, I have numpy version 1.23.5, my python3 version is 3.10.6.

The error i encounter is the following (however it should be noted that sometimes the error does not occur during a run, at other times it has occured hundreds of times during a run, and i can not see any patern in when it occurs. This is despite me using the same input data all the time):

Traceback (most recent call last):
  File "/home/carl/dev/emg_processing/src/./main.py", line 133, in <module>
    main(sys.argv[1:])
  File "/home/carl/dev/emg_processing/src/./main.py", line 120, in main
    classifier = ann(segment_arr, label_arr, args.k, args.dr, input_dim, args.l, args.sf, args.i, args.af, args.n, args.bs, args.nc, args.ns, args.ng, args.npm)
  File "/home/carl/dev/emg_processing/src/classifier/classifier.py", line 150, in ann
    best_solution_weights = ga(num_solutions, num_generations, num_parents_mating)
  File "/home/carl/dev/emg_processing/src/optimizer/optimizer.py", line 78, in ga
    ga_instance.run()
  File "/home/carl/.local/lib/python3.10/site-packages/pygad/pygad.py", line 1875, in run
    best_solution, best_solution_fitness, best_match_idx = self.best_solution(pop_fitness=self.last_generation_fitness)
  File "/home/carl/.local/lib/python3.10/site-packages/pygad/pygad.py", line 1963, in best_solution
    best_match_idx = numpy.where(pop_fitness == numpy.max(pop_fitness))[0][0]
IndexError: index 0 is out of bounds for axis 0 with size 0

However i've been able to find that this issue is the result of the fitness function returning NaN, which in turn is the result of the categorical cross entropy returning NaN. I've checked the predictions and the labels provided to the cross categorical entropy and i can't find anything that would answer why this would result in NaN, to me it looks reasonable. So im wondering if anyone has encountered this aswell and if anyone knows how to fix it.

The following is the code for the GA, im using a simple 2 layer ANN. My data has 11 possible classes. All 'config.' are imported global variables since fitness function only accepts three specific arguments.

import pygad.kerasga
import numpy
import tensorflow.keras

import config


#==============================================================================
# Fitness function for calculating the fitness of a solution
def fitness_func(ga_instance, solution, sol_idx):

    # Obtain the parameters from one solution
    model_weights_matrix = pygad.kerasga.model_weights_as_matrix(model=config.model, weights_vector=solution)

    # Use the solutions parameters to set the models parameters
    config.model.set_weights(weights=model_weights_matrix)

    # The model predicts on all the data
    predictions = config.model.predict(config.ga_data, verbose = 0)

    # We check how well it predicted and base it's fitness on how good it predicted
    cce = tensorflow.keras.losses.CategoricalCrossentropy()
    # Add the small value so we don't divide by 0
    solution_fitness = 1.0 / (cce(config.ga_labels, predictions).numpy() + 0.00000001)

    # Return the fitness
    return solution_fitness
#==============================================================================
# Used for debugging, tells the progress of the GA
def callback_generation(ga_instance):
    print("Generation = {generation}".format(generation=ga_instance.generations_completed))
    print("Fitness    = {fitness}".format(fitness=ga_instance.best_solution()[1]))
#==============================================================================
# Genetic Algorithm for optimization of classifier parameters
def ga(num_solutions, num_generations, num_parents_mating):

    # Which model to optimize and the number of soultions in the population
    keras_ga = pygad.kerasga.KerasGA(model=config.model, num_solutions=num_solutions)

    # Cannot have more parents than solutions in the population
    if(num_parents_mating > num_solutions):
        print(f"Number of parents mating was set to", {num_solutions}, "(down from", {num_parents_mating}, "), because number of parents mating must be less than number of soultions in the population")
        num_parents_mating = num_solutions

    # Create the initial population
    initial_population = keras_ga.population_weights

    # Number of generations and parents mating in the GA
    # Initial population of the GA
    # Which fitness function the GA should use
    ga_instance = pygad.GA(num_generations=num_generations, num_parents_mating=num_parents_mating, initial_population=initial_population, fitness_func=fitness_func, on_generation=callback_generation)

    # Run the GA
    ga_instance.run()

    # Get best solution and parameters
    solution, solution_fitness, solution_idx = ga_instance.best_solution()
    best_solution_weights = pygad.kerasga.model_weights_as_matrix(model=config.model, weights_vector=solution)

    # Return the best parameters found
    return best_solution_weights
#==============================================================================

The following is the predictions and labels (in that order) when the error occured after 194 generations:

array([[1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0.],
       [0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0.],
       [0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0.],
       [0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0.],
       [0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0.],
       [0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0.],
       [0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0.],
       [0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0.],
       [0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0.],
       [0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0.],
       [0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0.],
       [0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0.],
       [0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0.],
       [0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0.],
       [0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0.],
       [0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0.],
       [0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0.],
       [0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0.],
       [0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0.],
       [0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0.],
       [0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0.],
       [0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0.],
       [0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0.],
       [1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0.],
       [0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0.],
       [0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0.],
       [0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0.],
       [0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0.],
       [0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0.],
       [0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0.],
       [0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0.],
       [0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0.],
       [0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0.],
       [0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0.],
       [0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0.],
       [0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0.],
       [0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0.],
       [0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0.],
       [0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0.],
       [0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0.],
       [0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0.],
       [0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0.],
       [0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0.],
       [0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0.]])
array([[1.0000000e+00, 0.0000000e+00, 1.0000000e+00, 0.0000000e+00,
        1.6537365e-21, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00,
        0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
       [1.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00,
        0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00,
        0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
       [1.0000000e+00, 0.0000000e+00, 1.0000000e+00, 0.0000000e+00,
        0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00,
        0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
       [1.0000000e+00, 0.0000000e+00, 1.0000000e+00, 0.0000000e+00,
        1.3163597e-26, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00,
        9.7521068e-33, 0.0000000e+00, 0.0000000e+00],
       [0.0000000e+00, 0.0000000e+00, 1.0000000e+00, 1.0000000e+00,
        1.0000000e+00, 1.0000000e+00, 1.0000000e+00, 1.0000000e+00,
        0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
       [0.0000000e+00, 0.0000000e+00, 1.0000000e+00, 1.0000000e+00,
        1.0000000e+00, 1.0000000e+00, 1.0000000e+00, 1.0000000e+00,
        0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
       [0.0000000e+00, 1.6338916e-08, 1.0000000e+00, 1.0000000e+00,
        1.0000000e+00, 9.9997479e-01, 1.0000000e+00, 0.0000000e+00,
        0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
       [0.0000000e+00, 1.0000000e+00, 1.0000000e+00, 1.0000000e+00,
        1.0000000e+00, 1.5995245e-03, 1.0000000e+00, 0.0000000e+00,
        0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
       [0.0000000e+00, 7.4395859e-03, 1.0000000e+00, 1.0000000e+00,
        1.0000000e+00, 9.6971518e-01, 1.0000000e+00, 0.0000000e+00,
        0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
       [0.0000000e+00, 6.5632437e-15, 1.0000000e+00, 1.0000000e+00,
        1.0000000e+00, 1.0000000e+00, 1.0000000e+00, 5.8124778e-28,
        0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
       [0.0000000e+00, 4.5765560e-22, 1.0000000e+00, 1.0000000e+00,
        1.0000000e+00, 1.0000000e+00, 1.0000000e+00, 1.0761170e-25,
        0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
       [0.0000000e+00, 8.3525886e-12, 1.0000000e+00, 1.0000000e+00,
        1.0000000e+00, 1.0000000e+00, 1.0000000e+00, 0.0000000e+00,
        0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
       [0.0000000e+00, 3.5442305e-20, 1.0000000e+00, 1.0000000e+00,
        1.0000000e+00, 1.0000000e+00, 1.0000000e+00, 5.3136361e-08,
        0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
       [0.0000000e+00, 1.6981202e-15, 1.0000000e+00, 1.0000000e+00,
        1.0000000e+00, 1.0000000e+00, 1.0000000e+00, 2.0167827e-38,
        0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
       [0.0000000e+00, 9.9991322e-01, 1.0000000e+00, 1.0000000e+00,
        1.0000000e+00, 5.0599866e-02, 1.0000000e+00, 0.0000000e+00,
        0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
       [1.1111567e-27, 1.0000000e+00, 4.1279522e-01, 1.0000000e+00,
        1.0000000e+00, 1.3650597e-17, 1.0000000e+00, 0.0000000e+00,
        0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
       [0.0000000e+00, 7.4416973e-23, 1.0000000e+00, 1.0000000e+00,
        1.0000000e+00, 1.0000000e+00, 1.0000000e+00, 7.1864803e-09,
        0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
       [0.0000000e+00, 8.9365212e-32, 1.0000000e+00, 1.0000000e+00,
        1.0000000e+00, 1.0000000e+00, 1.0000000e+00, 1.0000000e+00,
        0.0000000e+00, 0.0000000e+00, 3.1516618e-38],
       [0.0000000e+00, 7.4043003e-28, 1.0000000e+00, 1.0000000e+00,
        1.0000000e+00, 1.0000000e+00, 1.0000000e+00, 1.0000000e+00,
        0.0000000e+00, 1.5207636e-37, 5.2818956e-35],
       [0.0000000e+00, 0.0000000e+00, 1.0000000e+00, 1.0000000e+00,
        1.0000000e+00, 1.0000000e+00, 1.0000000e+00, 1.0000000e+00,
        0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
       [0.0000000e+00, 1.0324602e-37, 1.0000000e+00, 1.0000000e+00,
        1.0000000e+00, 1.0000000e+00, 1.0000000e+00, 1.0000000e+00,
        0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
       [2.0097023e-15, 9.9998254e-01, 1.0000000e+00, 1.0000000e+00,
        1.0000000e+00, 2.5984167e-12, 1.0000000e+00, 0.0000000e+00,
        1.4740964e-32, 4.5718017e-33, 0.0000000e+00],
       [1.6984066e-36, 1.0000000e+00, 1.0000000e+00, 1.0000000e+00,
        1.0000000e+00, 1.6022282e-08, 1.0000000e+00, 0.0000000e+00,
        0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
       [0.0000000e+00, 4.3199220e-12, 1.0000000e+00, 1.0000000e+00,
        1.0000000e+00, 9.9999988e-01, 1.0000000e+00, 1.1980822e-21,
        0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
       [0.0000000e+00, 1.3270545e-13, 1.0000000e+00, 1.0000000e+00,
        1.0000000e+00, 9.9999976e-01, 9.9999970e-01, 1.8867243e-11,
        0.0000000e+00, 7.4753842e-38, 0.0000000e+00],
       [0.0000000e+00, 7.8818629e-29, 1.0000000e+00, 1.0000000e+00,
        1.0000000e+00, 1.0000000e+00, 1.0000000e+00, 1.0000000e+00,
        0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
       [0.0000000e+00, 0.0000000e+00, 1.0000000e+00, 1.0000000e+00,
        1.0000000e+00, 1.0000000e+00, 1.0000000e+00, 1.0000000e+00,
        0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
       [0.0000000e+00, 0.0000000e+00, 1.0000000e+00, 1.0000000e+00,
        1.0000000e+00, 1.0000000e+00, 1.0000000e+00, 1.0000000e+00,
        0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
       [0.0000000e+00, 0.0000000e+00, 1.0000000e+00, 1.0000000e+00,
        1.0000000e+00, 1.0000000e+00, 1.0000000e+00, 1.0000000e+00,
        0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
       [0.0000000e+00, 1.5401451e-16, 1.0000000e+00, 1.0000000e+00,
        1.0000000e+00, 1.0000000e+00, 1.0000000e+00, 0.0000000e+00,
        0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
       [0.0000000e+00, 1.0000000e+00, 1.0000000e+00, 1.0000000e+00,
        1.0000000e+00, 1.0748938e-07, 1.0000000e+00, 0.0000000e+00,
        0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
       [0.0000000e+00, 9.9751586e-01, 1.0000000e+00, 1.0000000e+00,
        1.0000000e+00, 1.1370505e-01, 1.0000000e+00, 0.0000000e+00,
        0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
       [0.0000000e+00, 0.0000000e+00, 1.0000000e+00, 1.0000000e+00,
        1.0000000e+00, 1.0000000e+00, 1.0000000e+00, 1.0000000e+00,
        0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
       [0.0000000e+00, 0.0000000e+00, 1.0000000e+00, 1.0000000e+00,
        1.0000000e+00, 1.0000000e+00, 1.0000000e+00, 1.0000000e+00,
        0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
       [0.0000000e+00, 0.0000000e+00, 1.0000000e+00, 1.0000000e+00,
        1.0000000e+00, 1.0000000e+00, 1.0000000e+00, 1.0000000e+00,
        0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
       [0.0000000e+00, 1.6451079e-16, 1.0000000e+00, 1.0000000e+00,
        1.0000000e+00, 1.0000000e+00, 1.0000000e+00, 1.1492193e-32,
        0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
       [0.0000000e+00, 2.3176689e-28, 1.0000000e+00, 1.0000000e+00,
        1.0000000e+00, 1.0000000e+00, 1.0000000e+00, 1.0000000e+00,
        0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
       [0.0000000e+00, 0.0000000e+00, 1.0000000e+00, 1.0000000e+00,
        1.0000000e+00, 1.0000000e+00, 1.0000000e+00, 1.0000000e+00,
        0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
       [0.0000000e+00, 0.0000000e+00, 1.0000000e+00, 1.0000000e+00,
        1.0000000e+00, 1.0000000e+00, 1.0000000e+00, 1.0000000e+00,
        0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
       [0.0000000e+00, 2.1104829e-32, 1.0000000e+00, 1.0000000e+00,
        1.0000000e+00, 1.0000000e+00, 1.0000000e+00, 1.0000000e+00,
        0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
       [0.0000000e+00, 1.5236378e-06, 1.0000000e+00, 1.0000000e+00,
        1.0000000e+00, 9.9994671e-01, 1.0000000e+00, 0.0000000e+00,
        0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
       [0.0000000e+00, 3.8478177e-25, 1.0000000e+00, 1.0000000e+00,
        1.0000000e+00, 1.0000000e+00, 1.0000000e+00, 1.0000000e+00,
        0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
       [0.0000000e+00, 2.5574896e-22, 1.0000000e+00, 1.0000000e+00,
        1.0000000e+00, 1.0000000e+00, 9.9997914e-01, 1.0000000e+00,
        0.0000000e+00, 5.1517299e-36, 6.5897658e-36],
       [0.0000000e+00, 2.7718766e-28, 1.0000000e+00, 1.0000000e+00,
        1.0000000e+00, 1.0000000e+00, 1.0000000e+00, 1.0000000e+00,
        0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
       [0.0000000e+00, 1.0999998e-23, 1.0000000e+00, 1.0000000e+00,
        1.0000000e+00, 1.0000000e+00, 1.0000000e+00, 9.9995071e-01,
        0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
       [0.0000000e+00, 0.0000000e+00, 1.0000000e+00, 1.0000000e+00,
        1.0000000e+00, 1.0000000e+00, 1.0000000e+00, 1.0000000e+00,
        0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
       [0.0000000e+00, 0.0000000e+00, 1.0000000e+00, 1.0000000e+00,
        1.0000000e+00, 1.0000000e+00, 1.0000000e+00, 1.0000000e+00,
        0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
       [0.0000000e+00, 1.4310640e-36, 1.0000000e+00, 1.0000000e+00,
        1.0000000e+00, 1.0000000e+00, 1.0000000e+00, 1.0000000e+00,
        0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
       [0.0000000e+00, 6.3025513e-10, 1.0000000e+00, 9.9999994e-01,
        1.0000000e+00, 1.0000000e+00, 1.0000000e+00, 1.0000000e+00,
        0.0000000e+00, 2.3318755e-21, 9.4422745e-08],
       [0.0000000e+00, 1.1929870e-15, 1.0000000e+00, 1.0000000e+00,
        1.0000000e+00, 1.0000000e+00, 1.0000000e+00, 1.0000000e+00,
        0.0000000e+00, 1.8064725e-25, 1.4417658e-13],
       [0.0000000e+00, 9.9369687e-01, 1.0000000e+00, 1.0000000e+00,
        1.0000000e+00, 3.2406049e-08, 1.0000000e+00, 0.0000000e+00,
        0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
       [0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00,
        0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00,
        0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
       [0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00,
        0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00,
        0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
       [0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00,
        0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00,
        0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
       [1.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00,
        0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00,
        0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
       [1.0000000e+00, 0.0000000e+00, 1.0000000e+00, 0.0000000e+00,
        1.6537365e-21, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00,
        0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
       [1.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00,
        0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00,
        0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
       [1.0000000e+00, 0.0000000e+00, 1.0000000e+00, 0.0000000e+00,
        0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00,
        0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
       [0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00,
        0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00,
        0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
       [1.0000000e+00, 0.0000000e+00, 1.0000000e+00, 0.0000000e+00,
        1.6537365e-21, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00,
        0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
       [0.0000000e+00, 7.4395859e-03, 1.0000000e+00, 1.0000000e+00,
        1.0000000e+00, 9.6971518e-01, 1.0000000e+00, 0.0000000e+00,
        0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
       [0.0000000e+00, 7.4395859e-03, 1.0000000e+00, 1.0000000e+00,
        1.0000000e+00, 9.6971518e-01, 1.0000000e+00, 0.0000000e+00,
        0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
       [0.0000000e+00, 7.4395440e-03, 1.0000000e+00, 1.0000000e+00,
        1.0000000e+00, 9.6971536e-01, 1.0000000e+00, 0.0000000e+00,
        0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
       [0.0000000e+00, 7.4395440e-03, 1.0000000e+00, 1.0000000e+00,
        1.0000000e+00, 9.6971536e-01, 1.0000000e+00, 0.0000000e+00,
        0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
       [0.0000000e+00, 7.4395859e-03, 1.0000000e+00, 1.0000000e+00,
        1.0000000e+00, 9.6971518e-01, 1.0000000e+00, 0.0000000e+00,
        0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
       [0.0000000e+00, 7.4395859e-03, 1.0000000e+00, 1.0000000e+00,
        1.0000000e+00, 9.6971518e-01, 1.0000000e+00, 0.0000000e+00,
        0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
       [0.0000000e+00, 7.4395859e-03, 1.0000000e+00, 1.0000000e+00,
        1.0000000e+00, 9.6971518e-01, 1.0000000e+00, 0.0000000e+00,
        0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
       [0.0000000e+00, 7.4395859e-03, 1.0000000e+00, 1.0000000e+00,
        1.0000000e+00, 9.6971518e-01, 1.0000000e+00, 0.0000000e+00,
        0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
       [0.0000000e+00, 7.4395859e-03, 1.0000000e+00, 1.0000000e+00,
        1.0000000e+00, 9.6971518e-01, 1.0000000e+00, 0.0000000e+00,
        0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
       [0.0000000e+00, 7.4395859e-03, 1.0000000e+00, 1.0000000e+00,
        1.0000000e+00, 9.6971518e-01, 1.0000000e+00, 0.0000000e+00,
        0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
       [0.0000000e+00, 7.4395859e-03, 1.0000000e+00, 1.0000000e+00,
        1.0000000e+00, 9.6971518e-01, 1.0000000e+00, 0.0000000e+00,
        0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
       [0.0000000e+00, 0.0000000e+00, 1.0000000e+00, 1.0000000e+00,
        1.0000000e+00, 1.0000000e+00, 1.0000000e+00, 1.0000000e+00,
        0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
       [1.0000000e+00, 0.0000000e+00, 1.0000000e+00, 0.0000000e+00,
        1.3163597e-26, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00,
        9.7521068e-33, 0.0000000e+00, 0.0000000e+00],
       [0.0000000e+00, 9.9991322e-01, 1.0000000e+00, 1.0000000e+00,
        1.0000000e+00, 5.0599866e-02, 1.0000000e+00, 0.0000000e+00,
        0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
       [0.0000000e+00, 0.0000000e+00, 1.0000000e+00, 1.0000000e+00,
        1.0000000e+00, 1.0000000e+00, 1.0000000e+00, 1.0000000e+00,
        0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
       [2.0097023e-15, 9.9998254e-01, 1.0000000e+00, 1.0000000e+00,
        1.0000000e+00, 2.5984167e-12, 1.0000000e+00, 0.0000000e+00,
        1.4740964e-32, 4.5718017e-33, 0.0000000e+00],
       [0.0000000e+00, 1.5236378e-06, 1.0000000e+00, 1.0000000e+00,
        1.0000000e+00, 9.9994671e-01, 1.0000000e+00, 0.0000000e+00,
        0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
       [0.0000000e+00, 0.0000000e+00, 1.0000000e+00, 1.0000000e+00,
        1.0000000e+00, 1.0000000e+00, 1.0000000e+00, 1.0000000e+00,
        0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
       [0.0000000e+00, 1.3270545e-13, 1.0000000e+00, 1.0000000e+00,
        1.0000000e+00, 9.9999976e-01, 9.9999970e-01, 1.8867243e-11,
        0.0000000e+00, 7.4753842e-38, 0.0000000e+00],
       [0.0000000e+00, 1.3270545e-13, 1.0000000e+00, 1.0000000e+00,
        1.0000000e+00, 9.9999976e-01, 9.9999970e-01, 1.8867243e-11,
        0.0000000e+00, 7.4753842e-38, 0.0000000e+00],
       [0.0000000e+00, 1.3270545e-13, 1.0000000e+00, 1.0000000e+00,
        1.0000000e+00, 9.9999976e-01, 9.9999970e-01, 1.8867243e-11,
        0.0000000e+00, 7.4753842e-38, 0.0000000e+00],
       [0.0000000e+00, 1.3270545e-13, 1.0000000e+00, 1.0000000e+00,
        1.0000000e+00, 9.9999976e-01, 9.9999970e-01, 1.8867243e-11,
        0.0000000e+00, 7.4753842e-38, 0.0000000e+00],
       [0.0000000e+00, 1.5236378e-06, 1.0000000e+00, 1.0000000e+00,
        1.0000000e+00, 9.9994671e-01, 1.0000000e+00, 0.0000000e+00,
        0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
       [0.0000000e+00, 1.5236378e-06, 1.0000000e+00, 1.0000000e+00,
        1.0000000e+00, 9.9994671e-01, 1.0000000e+00, 0.0000000e+00,
        0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
       [0.0000000e+00, 1.3270545e-13, 1.0000000e+00, 1.0000000e+00,
        1.0000000e+00, 9.9999976e-01, 9.9999970e-01, 1.8867243e-11,
        0.0000000e+00, 7.4753842e-38, 0.0000000e+00],
       [0.0000000e+00, 2.5574896e-22, 1.0000000e+00, 1.0000000e+00,
        1.0000000e+00, 1.0000000e+00, 9.9997914e-01, 1.0000000e+00,
        0.0000000e+00, 5.1517299e-36, 6.5897658e-36],
       [0.0000000e+00, 1.1929870e-15, 1.0000000e+00, 1.0000000e+00,
        1.0000000e+00, 1.0000000e+00, 1.0000000e+00, 1.0000000e+00,
        0.0000000e+00, 1.8064725e-25, 1.4417658e-13],
       [0.0000000e+00, 2.1104829e-32, 1.0000000e+00, 1.0000000e+00,
        1.0000000e+00, 1.0000000e+00, 1.0000000e+00, 1.0000000e+00,
        0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
       [0.0000000e+00, 2.1104829e-32, 1.0000000e+00, 1.0000000e+00,
        1.0000000e+00, 1.0000000e+00, 1.0000000e+00, 1.0000000e+00,
        0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
       [0.0000000e+00, 2.1104829e-32, 1.0000000e+00, 1.0000000e+00,
        1.0000000e+00, 1.0000000e+00, 1.0000000e+00, 1.0000000e+00,
        0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
       [0.0000000e+00, 2.1104829e-32, 1.0000000e+00, 1.0000000e+00,
        1.0000000e+00, 1.0000000e+00, 1.0000000e+00, 1.0000000e+00,
        0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
       [0.0000000e+00, 1.1929870e-15, 1.0000000e+00, 1.0000000e+00,
        1.0000000e+00, 1.0000000e+00, 1.0000000e+00, 1.0000000e+00,
        0.0000000e+00, 1.8064725e-25, 1.4417658e-13],
       [0.0000000e+00, 2.1104829e-32, 1.0000000e+00, 1.0000000e+00,
        1.0000000e+00, 1.0000000e+00, 1.0000000e+00, 1.0000000e+00,
        0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
       [0.0000000e+00, 1.1929870e-15, 1.0000000e+00, 1.0000000e+00,
        1.0000000e+00, 1.0000000e+00, 1.0000000e+00, 1.0000000e+00,
        0.0000000e+00, 1.8064725e-25, 1.4417658e-13],
       [0.0000000e+00, 1.1929825e-15, 1.0000000e+00, 1.0000000e+00,
        1.0000000e+00, 1.0000000e+00, 1.0000000e+00, 1.0000000e+00,
        0.0000000e+00, 1.8064863e-25, 1.4417685e-13],
       [0.0000000e+00, 2.1104829e-32, 1.0000000e+00, 1.0000000e+00,
        1.0000000e+00, 1.0000000e+00, 1.0000000e+00, 1.0000000e+00,
        0.0000000e+00, 0.0000000e+00, 0.0000000e+00]], dtype=float32)

Answer 1

I had the same issue of receiving NaN in the prediction when using a CNN with batch normalization and dropout. After removing that, the issue was resolved.