OpenSlide with TCGA images. Some images are with 40x zoom and some with 20x. How can I get the tiles from DeepZoomGenerator be all at the same scale?

Question

Working with the TCGA breast cancer dataset, I'm attemting to have all of the tiles saved be from a 20x zoom image as opposed to a mix of 20x and 40x. My model is struggling to push past a threshold that I think it should be able to reach but it's overfitting. I'm concerned that my model is struggling to learn because tiles with both zoom levels are in the training data making things be inconsistant.

I have been working on the following code to try and aleviate the issue:

def save_loop(row1, col1, directory, tiles, imagename, proc, levelnum):
    for row in row1:
        for col in col1:
            tile_name = os.path.join(directory, imagename + '_%d_%d' % (col, row))
            # print("Now saving tile with title: ", tile_name)
            temp_tile = tiles.get_tile(levelnum, (col, row))
            temp_tile_np = np.array(temp_tile.convert('RGB'))
            if temp_tile_np.mean() < 170 and temp_tile_np.std() > 15 and temp_tile_np.shape == (256, 256, 3):
                plt.imsave(tile_name + ".png", temp_tile_np)

for index, row in images_to_process.iterrows():
    tile_dir = f"{SLIDES_PATH}/{row.id}/tiles"
    # Load the slide file (svs) into an object.
    slide = open_slide(f"{SLIDES_PATH}/{row.id}/{row.filename}")
    objective = float(slide.properties[openslide.PROPERTY_NAME_OBJECTIVE_POWER])
    tiles = DeepZoomGenerator(slide, tile_size=256, overlap=0, limit_bounds=False)
    if objective == 20:
        level_num = len(tiles.level_tiles) - 1
        dim_lev = 2
    else:
        level_num = len(tiles.level_tiles) - 2
        dim_lev = 3

    level4_img = slide.read_region((0, 0), dim_lev, level4_dim)  # Pillow object, mode=RGBA
    # Convert the image to RGB
    level4_img_RGB = level4_img.convert('RGB')
    plt.imshow(level4_img_RGB)
    plt.show()
    
    cols, rows = tiles.level_tiles[level_num]
    save_loop(rows, cols, tile_dir, tiles,row.filename[:-4],level_num)

I'm expecting the tile level_num to be pulling from the downsampled level (len(tiles.level_tiles) - 2 = zoomed out??) image if objective is 40.

Am I thinking this through correctly?