Use TPL5110 for power management
The current setup uses multiple mA in standby.
Use a TPL5110, which should bring this down to the μA range.
https://www.ti.com/lit/ds/symlink/tpl5110.pdf
The TPL5110 should switch a MOSFET, Adafruit uses a DMG3415: