Skip to content

Build manylinux wheels for CI pipeline tests/doc building instead of using conda environments

finesse importer requested to merge ci-tweaks into master

Closes #218 (closed).

Our use of conda was causing issues with the CI pipeline because the artefacts passed between jobs (the conda environments) were getting too large for the servers to handle reliably. We decided we don't need conda for any of the CI pipeline, since all of Finesse's dependencies (for building, testing and docs) are available via PyPI and package managers in various Docker images. One complication for Finesse 3 is its use of Cython extensions and the SuiteSparse system (not Python) dependency. We figured out how to get these to play nicely with bdist wheels in in #218 (closed). This MR implements the result in the pipeline so that:

  • bdist wheels are built for Python 3.8 and 3.9 in manylinux form
    • these are currently stored as artefacts that GitLab deletes after a while
    • once we reach the point that we want to start sharing Finesse, we should add a job that uploads these wheels to PyPI
  • tests are applied using the test suite in the current branch's HEAD but they are applied to the wheel the job gets passed from the build stage
    • this is good practice, since it tests the real distributions that would be provided to users, and avoids potential issues with installed repository files appearing on the PATH and making tests pass/fail when they shouldn't/should, and catches cases where production dependencies are mis-listed as dev-only dependencies
    • tests are run on clean, standard Debian-based Python 3.x images (but we could technically run it on any Linux we want)
  • docs are built using the docstrings in the actual wheel
    • again, we test the real release version, which lets us catch more bugs

Changes from current master

I removed the BEST testing stage. I got the impression from Aaron that this stuff is soon to be merged into the core repository anyway, so I jumped the gun. If that was premature we can add the stage back in.

Running make html or make pdf or whatever for the documentation will no longer automatically build Finesse itself. This was causing problems because the makefiles had hard-coded python3 commands that did or didn't exist and point to the same Python version across various Docker image flavours. I think it anyway makes sense for doc building to not have side-effects like also building Finesse itself. Like BEST, this can be restored with some effort if someone disagrees with this removal.

Other notes

Building on Mac OSX should be possible, since git.ligo.org provides Sierra and High Sierra runners, but I haven't got the slightest clue how to go about getting SuiteSparse and GCC and whatnot on there so I'll leave that to one of the Mac users. If someone can provide me with a bunch of bash commands to run to get this part working, I can implement another CI job to do the equivalent building and testing on OSX.

Windows is going to be another issue. There we apparently need to install Microsoft Visual C to get a C compiler. There's also no BASH as far as I am aware. Again, if someone knows what to do there, I can set up the CI (once we have a Windows runner at Nikhef).

Edited by finesse importer

Merge request reports