Build CSG files as intermediate steps
Currently we build the STLs directly from OpenSCAD source. It's hard to determine which STLs have changed based on the changes in the OpenSCAD source, because there are a lot of dependencies - and changing a dependency doesn't always result in a change in geometry. Adding an intermediate build step, where we first compile OpenSCAD to CSG (which is just OpenSCAD but with all the includes resolved and variables substituted for constants), makes it much clearer what has changed. For example, changing a module in a dependency that isn't used will not change the CSG file, and thus can be ignored.
This solves two problems:
- The build is slow - knowing which STLs to rebuild will speed it up massively, if we can cache the built STLs.
- It's hard to tell which STL files have changed between releases (OpenSCAD builds STLs non-deterministically so often an output STL will change even if the geometry is identical).
If we build (and save hashes of) the CSG files, we could:
- Only build STLs that have actually changed (incremental builds)
- Maintain a list of STLs that have changed (or not) from version to version - even if we rebuild all the STLs it would be helpful to know which parts have changed.
- With a bit more work (mostly splitting out the camera information from
render.py) we could even regenerate the renders incrementally.
That should save an enormous amount of build time. However, it does seem to run contrary to the CI philosophy of "regenerate everything from scratch every time" so I would welcome feedback on how wise this is, particularly from @julianstirling and @kasbah. My conclusion is that, so long as we are careful to hash everything robustly, it shouldn't be a problem to use a cache.
This MR will:
-
Generate CSG files as an intermediate step in the build process (OpenSCAD -> CSG -> STL) -
Process those CSG files so they don't contain timestamps -
Calculate hashes of all the CSG files so they can be compared to previous builds -
cache the corresponding STL files somewhere (e.g use CI cacheing) -
Pay attention to the dependencies of those STL files (CSG files don't useorincludebut can stillimport).-
Figure out how to take account of STL (or other) dependencies in the hashing process, probably with more checksums
-
Tasks left for the future:
- Think about renders - we'd need to include the camera variables somehow (these could reasonably be embedded into the csg file post-hoc: it wouldn't then be as simple as a CSG file exported from OpenSCAD, but it would still be valid OpenSCAD input). This would have the pleasing consequence that we embed all the information in the CSG file, which is hashed, i.e. we don't need additional metadata in the cache system.
I suspect many of those tasks will move to another issue/MR, if it proves that just generating the CSG files is a useful intermediate stage.
NB I have worked around an OpenSCAD issue (I think) with relative paths for CSG export, which is why CSG files are specified with absolute paths when they are exported.