imshow uses a seemingly random background color

Question

I am trying to use imshow(), with categorical data. But I cannot reliably control the colors used. This is my code (inspired in the solution provided here

from random import choice
import matplotlib.pyplot as plt # draw pretty stuff
from matplotlib import colors

def draw_trace2(mem_trace):
    print(f"Matrix: {mem_trace.block_size} rows, {mem_trace.len()} cols")
    # this prints: 'Matrix: 80 rows, 178 cols'

    dummy = True
    if dummy:
        # create dummy data
        access_matrix = [
            [choice([10,10,10,10,20,30]) for x in range(mem_trace.len())]
            for y in range(mem_trace.block_size)]
    else:
        # create real data
        access_matrix = [
            [10 for x in range(mem_trace.len())]
            for y in range(mem_trace.block_size)]
        for ac in mem_trace.access_list:
            for i in range(ac.size):
                access_matrix[ac.offset+i][ac.time] = 20 if ac.action=='R' else 30

    # paranoically check correct values
    for row in access_matrix:
        for cell in row:
            if cell not in [10,20,30]:
                raise ValueError("Wrong Value")

    # create discrete colormap. Credit to the previously cited SO answer
    cmap = colors.ListedColormap(['white', 'green', 'red'])
    bounds = [0,15,25,35]
    norm = colors.BoundaryNorm(bounds, cmap.N)

    # draw and export image
    fig, axe1 = plt.subplots()
    axe1.imshow(access_matrix, cmap=cmap, norm=norm)
    fig.set_size_inches(100,44)
    fig.savefig('mem_trace_plot.pdf', bbox_inches='tight')
    return

• dummy=True: I get dummy data with a red background.
• dummy=False: I get the real data, with what looks to be 50% gray background or sometimes red too

My intention is to have white background. Another oddity: If I reduce the size of the plot (for example, fig.set_size_inches(10,4.4)) the problem goes away.