Currently, the preview shown while the microscope is scanning is a JPEG image, generated by running `openflexure-stitching` in a background process. This is a inefficient, as we end up regenerating a large image quite frequently - that puts strain on limited CPU/memory resources, and means the preview usually lags behind image acquisition. It would be possible to implement a preview using WebGL instead. This wouldn't have to do any image registration: that's already done on the server using a background process (separate to the one that generates the preview image). Moving the preview to WebGL would shift some load from the Raspberry Pi to whatever computer is showing the interface. It would potentially also be faster as there's no need to write a large image to a compressed JPEG and re-inflate it. It would be nice to explore the use of WebGL; if it looks like it delivers UI/performance improvements without too high a maintenance overhead, this would be a candidate for merging into the web app. Prototyping this in a student project would be a good way to determine if it's a good idea to include. Some specifics: In order to show a preview, we need to display a number (perhaps several hundred) of images on a WebGL canvas. These images will overlap, see below for how we pick which image is on top. The code to position the images (based on the stage coordinates and correlations between overlapping images) already runs on the server, so we'll need to add a couple of endpoints to: * Serve the current list of images and positions. Could be a dictionary mapping filenames to coordinates. * Serve the images, at very much reduced resolution. Ideally we'd do some sort of mipmap, so we'd be able to request the images at full resolution, half, quarter, and so on until they are tiny. It may be that it makes sense to use a factor of 4 rather than 2, this is just a suggestion. Those should be fairly simple to implement (and might be done by someone familiar with the server to save the student learning two codebases). Once they exist, the main bit of this project will be implementing the WebGL renderer. It will need to: * Create a canvas within the [web app]. * Retrieve the list of images and coordinates from the server (see above). * Download each image at low resolution, and use it as a texture. Render the texture on a rectangular shape, with the correct position. That's enough for it to achieve feature parity with the current code. However, a WebGL implementation could go further, for example by supporting scrolling/dragging/zooming of the viewport. Zooming in would work much better if it was able to dynamically swap to higher-resolution textures of the images currently being viewed: it's that last part that I anticipate will be most challenging, as it requires careful memory management. There are existing solutions like [openseadragon] that do this very nicely. The one thing I think they lack, though, is the ability to handle overlapping images nicely. I'd love to discover that overlapping images do work, in which case what we probably want to do is figure out a way to use [openseadragon] to show our preview. There is, I think, a useful trick that would make it much easier to display overlapping images in the way we currently do. Our current algorithm takes each pixel value from exactly one image, and that's the image whose centre is closest to the current position. This means we're preferring the centres of the images, which are where the resolution and colour fidelity are best. Happily, this can be implemented without lots of complicated distance calculations, if we render in 3D. The obvious way to render the scan preview would be to render each image as a flat quadrilateral. However, if we render each image as being slightly curved, with the middle closer to the camera than the edges, we'll always see the image that's closest, i.e. the image that has its centre closest to that position. I'll paste in a couple of pictures to show the concept, but this is something we can discuss once we get to it.   This should be a fun project for a student that knows some OpenGL and Javascript. While we can help with the OpenFlexure-specific stuff, there's not really capacity within the team to train someone up on WebGL as it's a new technology to use as well. [web app]: https://gitlab.com/openflexure/openflexure-microscope-server/-/tree/v3/webapp?ref_type=heads [openseadragon]: https://openseadragon.github.io/