Hard crashing the Pi

Bug report

Summary

The pi itself can crash fully from some errors!

Configuration

I'm using:

Pi:: Raspberry Pi 4B (2GB)

Camera: Raspberry Pi camera v2.1

Motor controller: Sangaboard

Steps to reproduce

This was using @jaknapper's v3_tissue branch modified to give full resolution images. The full images get Fourier Transformed, into objects that quickly eat the memory. Rather than getting a graceful memory error, the pi itself fully locks up. Note that full res images use about quarter of a GB of ram per cached FFT: openflexure-stitching#24 (closed)

Testing just in openflexure-stitching CLI on the Pi, when the RAM filled up terminal returned killed which implies that SIGKILL was used to kill the process. It seems that something to do with this running from LabThings/Systemd causes a full crash.

Relevant logs and/or screenshots

The main OFM log file and journalctl (looking at the server processes) show this happening at a random time because the stitching code doesn't regularly log and as it is in a thread the other processes get to different places each time before the crash.

@jaknapper We should check the kernel logs to check if there is anything in there next time we have a code session.

/cc @rwb27