How to prevent crash on STM32H7 when swapping flash banks

Question

I am using an STM32H753 which has two flash banks. For my firmware update I use the bank swapping feature so the mcu boots to the correct version.

When I swap the banks, the microcontroller crashes. No hardfault or any other exception. The debugger tells me that the PC is at 0x72c269e and the SP is at 0xbf00d7b4, but these have been different before and I assume those addresses are random.

Eventually the watchdog kicks in and resets the mcu after which the banks are swapped, so it did work. When I swap the banks back, there is no crash and everything works fine.

Here's the code that does this:

pub fn swap_bank(flash: &mut stm32h7xx_hal::stm32::FLASH) -> Result<(), &'static str> {
    assert!(flash.optcr().read().optlock().bit_is_clear());

    // Get the current config
    let current_bank = get_current_bank(flash) != 1;
    let swapped_bank = !current_bank;

    // Config the new bank
    flash
        .optsr_prg()
        .modify(|_, w| w.swap_bank_opt().bit(swapped_bank));

    // Start the config write
    flash.optcr().modify(|_, w| w.optstart().set_bit());
    
    // Wait for the change to propegate
    while flash.optsr_cur().read().opt_busy().bit_is_set() {
        cortex_m::asm::nop();
    }

    // We're done, now we need to reboot
    Ok(())
}

pub fn get_current_bank(flash: &mut stm32h7xx_hal::stm32::FLASH) -> u8 {
    flash.optcr().read().swap_bank().bit() as u8 + 1
}

If I put a bkpt instruction before the while loop, it gets hit both times (and if I then continue there's no crash both times).
If I put a bkpt instruction after the while loop, it does not get hit when swapping the first time.

The reference manual (4.3.13) tells us this:

1. Unlock OPTLOCK bit, if not already unlocked.
2. Set the new desired SWAP_BANK_OPT value in the FLASH_OPTSR_PRG register.
3. Start the option byte change sequence by setting the OPTSTART bit in the FLASH_OPTCR register.
4. Once the option byte change has completed, FLASH_OPTSR_CUR contains the expected SWAP_BANK_OPT value, but SWAP_BANK bit in FLASH_OPTCR has not yet been modified and the bank swapping is not yet effective.
5. Force a system reset or a POR. When the reset rises up, the bank swapping is effective (SWAP_BANK value updated in FLASH_OPTCR) and the new firmware shall be executed.

I think I am doing this correctly. I've also checked the errata and there were issues in older silicon with bank switching, but I'm using the newest V revision.

I hope someone can give me a tip.