sometimes a downgrade to ESP-IDF 4.4 is required to make wifi work again

I have observed a very strange issue for the second time now:

Sometimes (around 1 in 50 times, estimation based on two occurances) when I flash a new version of my application to my ESP32 chip, it will afterwards fail to connect to wifi. No amount of rebooting / re-flashing with ESP-IDF 5.1 will fix this, only by stepping back to ESP-IDF 4.4 does it connect to wifi again - afterwards I can upgrade to 5.1 again and it still works.

In that broken wifi-state neihter the client mode works, nor the AP-mode (using mixed config to have both). The AP name is visible initially after boot, but trying to connect to it from my laptop makes it vanish.

Can someone explain what might be going on here and maybe even have any ideas on how to prevent this? Should I maybe try and stick to ESP-IDF 4.4?

I'd like to be able to deploy my ESP32 chips in ways that don't allow reconnecting a serial adapter on around every 50th software update when wifi doesn't work anymore...