Use TPL5110 for power management

The current setup uses multiple mA in standby.

Use a TPL5110, which should bring this down to the μA range.

https://www.ti.com/lit/ds/symlink/tpl5110.pdf

The TPL5110 should switch a MOSFET, Adafruit uses a DMG3415:

https://learn.adafruit.com/assets/39783