Build dependencies for Windows via CI jobs

Following description is both a problem statement and a proposed idea for solution with some decisions to make.

Windows build is set up in a very customised way for OpenMW. The used dependencies for it are mostly pulled as binaries from different sources. Mostly they are stored in the openmw-deps. But how they are built is somewhat hidden knowledge. This is not good for an open source project. This should be open to the public information line on Linux where OpenMW is usually delivered as a package depending on other packages integrated into the distribution package manager. It's open and public knowledge how these packages are built. We need to set up the Windows build in a similar way.

MacOS potentially could be in the scope here. But I have no idea what is the current process for building dependencies and releasing OpenMW there.

There is no package manager on Windows providing all needed dependencies similar to Linux distributions. The idea is to use vcpkg with a OpenMW specific fork repository to build most of them. Basically the pipeline should contain following steps:

  1. Build all dependencies present in vcpkg and cache artefacts.
  2. Build other dependencies for whatever reasons not possible to have in vcpkg.
  3. Get other dependency binaries from trusted sources like Qt via aqt.
  4. Build openmw based on the dependencies above. This should work the same way in CI and for local builds.

The important condition for this to work is that there should be no transitive dependencies from different sources. For example OpenSceneGraph depends on COLLADA DOM which depends on Boost but OpenMW itself depends on Boost. In this setup the same Boost binary should be used for both.

A vcpkg repository fork is required to control what versions each dependency has and to adjust the build for OpenMW specifics. For example, the OpenSceneGraph version of the vcpkg does not fit OpenMW. It has to use the OSGoS fork.

There are multiple technical decisions to make:

  1. What CI system to use for building dependencies?
    • Gitlab
      • Pros
        • Same environment as OpenMW CI
      • Cons
        • Very limiting CI minutes budget (hard to iterate on start, easy to get into a situation not being able to build)
        • Slow Windows jobs
    • Github
      • Pros
        • Fast to run Windows jobs
        • No CI budget limits
      • Cons
        • Potential integration problems with CI on Gitlab
  2. Where to store build artefacts? (This is actually about both a type of storage and the platform)
    • Using CI integration
      • Pros
        • Simple to setup
      • Cons
        • Limited retention
        • Limited versioning (might be hard to bind OpenMW version with dependencies)
    • Git LFS repository
      • Pros
        • No retention limits
        • Advanced versioning (each version of OpenMW can be bound with specific dependencies version)
      • Cons
        • Non trivial to setup (need to give access to the CI job to push artefacts into Git repository potentially on different platform)
  3. Which packages to build using vcpkg? This is an open question but with initial suggestions.
    • Qt - no
    • OpenSceneGraph - if we can get OSGoS build with vcpkg.
    • Yes (not full list yet):
      • bullet3 (joltphysics)
      • boost-geometry
      • boost-iostreams
      • boost-locale
      • boost-program-options
      • collada-dom
      • freetype
      • ffmpeg
      • icu
      • libiconv
      • libjpeg-turbo (there is no libjpeg in the vcpkg)
      • libpng
      • libxml2
      • luajit
      • lz4
      • mygui (need support for MYGUI_DONT_USE_OBSOLETE option)
      • openal-soft
      • opengl-registry
      • sdl2
      • tiff
      • zlib
  4. Build static or dynamic libraries?
    • Dynamic
      • Pros
        • Keep current setup
      • Cons
        • Need to copy them around
    • Static
      • Pros
        • Less moving parts
      • Cons
        • Possible build scripts and integration issues
  5. Where to store CI config?
    • Separate repository
    • Main OpenMW repository
Edited by elsid