Looks like it has to do something with the bootloader.
I previously flashed the full factory-firmware which comes out of ESHome(platformio) build, but if I just flash the firmware to 0x10000, then it boots up all the time from battery as well. Needs more digging what could go wrong with platformio bootloader.