Draft: Version 3

This is the next major version of the microscope server. It is based on labthings-fastapi (will become Labthings 2), and focuses on making the server code cleaner, easier to maintain, and more secure. It also introduces performance enhancements that enable us to properly use websockets, and makes the code far more modular, enabling it to run on more diverse hardware. There are a lot of things to do to get this ready, which I'll try to detail below. We are already using it in anger for some of the pathology work Joe is leading, so features for this tend to get prioritised.

The summary below was updated as of mid-November 2024. It's not ordered very carefully yet, and is intended more for the developers than for anyone thinking of using it.

Architecture

We will use labthings-fastapi instead of labthings. This swaps Flask 1, Marshmallow and friends for FastAPI (Starlette, pydantic, etc.). Partly, this fixes a bunch of dependency rot, but it also streamlines the code hugely: most of the typing/schema generation can now be done with type hints. This eliminates vast reams of boilerplate code, which should be really helpful.

Extensions are no more: the only unit of code on the server is the Thing. There's a Thing for the camera, one for the stage, one for autofocus, and so on. This means there are more or less no "core" endpoints - so there's consistency between the built-in stuff and third party code. It also means we can split up the repo, for example most of the camera code is now in labthings-picamera2 and the stage wrapper is in labthings-sangaboard. This is done with a view to enabling much greater customisation of hardware.

Each Thing has a dictionary of settings, Thing.thing_settings that should be synced with a JSON file on disk. This is implemented, but manual synchronisation is needed at the moment.

Everything above is implemented already, although not all functionality from v2 of the server exists yet in v3.

The plan is to introduce a standard camera and stage interface, making it possible to reliably swap out cameras. This has been begun in !181 (merged) but it's likely it will evolve before it's ready to release. Better handling of Blob objects (things that can't be nicely serialised to JSON, like images) will enable us to have parity between the HTTP and Python APIs. That's implemented in labthings-fastapi#83, to be merged soon.

Configuration

The server v2 does a certain amount of autoconfiguration - for example, it will fall back to a dummy stage if the sangaboard isn't found. We have deliberately not done this for v3, and instead will provide a way to configure the server by specifying which Python object to use for each Thing on the server. This is implemented in !181 (merged) and should be merged in soon. The same MR makes it possible to fail to a fallback server, so that headless microscopes serve an error page, rather than failing to appear at all.

Having a defined configuration makes the code easier to test, and makes it more obvious when there's an error (e.g. you find out immediately if your stage is broken, not just when you try to move). However, it's not as friendly for people getting started. Our plan is to implement a management server, a separate web app that will be able to change the configuration (ideally with some auto-suggestions based on hardware scans), meaning you get an out-of-box experience that starts with a setup page or two, before starting a properly-configured server. The same management server might even be able to do other things - like restart the server, export settings, upgrade the software, start an iPython notebook server for programming, and so on. This second web app is not yet started, there's currently an empty repo.

Scanning

A major focus of our work is sample scanning for pathology applications. Very active development is happening on the blood_stack branch. The scanning code is now faster, easier to use, and more reliable - we have made huge progress here. However, it needs some major refactoring to tidy it up for maintainability and modularity. In particular, there are several chunks to factor out:

• Smart z stacking should be moved to its own Action, maybe on its own Thing. Needs the ability to return multiple images in its output, provided by labthings-fastapi#83
• Background detection should be in a separate module.
• Processing raw images belongs in the camera Thing, ideally with a defined API that enables camera swapping.
• Logic around feedforward focus control and path planning should be better organised, perhaps in objects. This may end up not being part of the HTTP API but should still get tidied up.

Known Issues

The intention is to document these with issues on GitLab, but it's worth listing a few here:

• Thing settings don't save automatically, and a "save_all_settings" action is used as a workaround.
• Scanning in blood_stack is optimised for blood smears and some settings are hard coded. This scanning code needs to be generalised and refactored.
• Scanning currently uses the camera Thing directly, bypassing the camera API and breaking alternative camera models.
• Running without a sangaboard is only supported by !181 (merged).
• The initial calibration wizard needs to be closed before you can focus the microscope, which is required for step 2. This needs work. That's an issue for v2 as well.

Building & Dependencies

The image customiser builds a Bookworm based OS image that includes v3. Currently, it uses the main branch of openflexure-stitching and the blood_stack branch of this repo. These branches and commit hashes are now noted in a log file alongside the image. All Python dependencies are installed via PyPI, though currently this is done by pip install -e . --only-binary=:all:, i.e. we don't use requirements.txt. Using pinned dependencies has advantages and disadvantages, it probably makes sense once there is better CI in this repo. We would most likely need separate requirements files for different platforms.

Currently, there are a few development platforms in use:

• Bookworm, 32 bit, Python 3.11, on Raspberry Pi 4 2Gb or 4Gb. This is where we deploy and do most dev work.
• Windows 11, 64 bit, Python 3.12. This is Richard's laptop and works in simulation mode since !181 (merged).
• Ubuntu 24 on WSL2, Python 3.12. Also a dev environment of Richard's.

Gitlab CI should probably just use default Python images and simulation mode. However, building a custom Docker container from our OS mage would let us test installation/dependencies. This would be useful, if slow.

There are now more repos involved than there were before:

• openflexure-stitching is installed in its own virtual environment, and called by the smart_scan Thing in a subprocess.
• openflexure-microscope-server is this repo, holding the top level server code, and several Things providing functionality (openflexure_microscope_server.things). It also includes the web app interface.
  • labthings-fastapi provides the infrastructure to serve a nice WoT compliant HTTP interface.
  • labthings-picamera2 wraps picamera2 (the official Raspberry Pi library) to provide calibration appropriate to a microscope and HTTP interface to it.
  • labthings-sangaboard exposes the Sangaboard stage to LabThings, wrapping
    • pysangaboard, the original Sangaboard library that is still used. It's possible we could drop this in the future, but that may not be desirable.
• openflexure-connect hasn't changed in some time, and there are just enough API v1 endpoints added to v3 that microscopes should still be detected and will still work.

Splitting out the hardware Things should help keep the code more modular, and make testing more feasible.

CI and Testing

LabThings-FastAPI is currently unit tested against several Python versions with >80% coverage, a figure that should increase. It's also checked with mypy and ruff. My intention is to do this with the server repo, and the other two labthings repos above - though testing labthings-picamera2 will need hardware. That's a big motivation for splitting it out: it means we can re-run those tests when the camera code changes, and avoid trying to test them when the server changes (which is much more frequent). Using a Docker image that replicates the Raspberry Pi environment would be helpful, as it will let us check dependency issues and some (though not all) system-level integration issues.

CI on this repo currently fails for the v3 branch, and this definitely needs to be fixed. In particular, enforcing type checking and code quality ought to mean that at least the main v3 branch stays a bit tidier.

Building the OS image is currently done manually, on a Glasgow server, as root, due to the need to manipulate disk image mounts. I don't believe this ever worked as anything more secure than a privileged Docker container, but I'd be delighted if someone figured out how to get this running in CI. It's probably not high priority, though.