Analysis of why tag pipelines take \~20 min and why `juice install --rebuild` is slow, with non-breaking levers to fix both. Measured 2026-09-28 against v26.33.0–v26.39.0 (GitLab API, job logs, registry manifests, `podman inspect`/`history`) unless marked as an estimate. ## Analysis Building the package isn't the bottleneck: the wheel itself builds in about 12 s. The time goes into two things: - **Re-uploading about 605 MB of image layers every release.** Most of that content is unchanged from the previous release, or is never used. - **CI jobs waiting on work they don't need.** Sites then download the same \~605 MB on every update, and redo system setup in every environment image. ### 1. Where the tag pipeline spends its time (v26.39.0: 1185 s, about 20 min) | Critical-path job | Time | Note | |-------------------|------|------| | tests | \~100 s | | | Package Python | 71 s | \~28 s of this is micromamba setup the job doesn't need | | Package RPM | 223 s | Only on the critical path because the render job needs it | | Render Base Image Context | 82 s | 52 s is `dnf install` of the RPM (it pulls in podman and more) to run a 3 s template render | | **Build CI Base Image** | **669 s** | Build \~144 s, **push 505 s** | | Release jobs | \~30 s | Wait for the image push | - The image job took 669–777 s on each of the last 7 tags. - Main, release-branch and scheduled pipelines take about 7–8 min and push no image. Their critical path is tests → Package Python → the slowest of RPM, DEB or docs (\~200–260 s each) → docs upload. ### 2. Why the push is slow: no layer is reused between releases I compared layer digests for `ci:26.33.0` through `ci:26.39.0`. Only the AlmaLinux layer is shared. Every other layer gets a new digest each release, even when its content is identical. | Layer | Compressed | Reused across releases? | |-------|------------|-------------------------| | almalinux:10.1 | 68 MB | yes | | `dnf install epel-release` | 3 MB | no | | `dnf install git nano nodejs mesa-libGL pandoc socat` | 248 MB | no (same size in 26.38.0 and 26.39.0, new digest) | | `COPY uv:latest` | 25 MB | no | | wheel, user setup, pyproject | \~1 MB | no (expected) | | `uv sync` | 327 MB | no | - Kaniko runs without `--cache`, and the build isn't reproducible, so identical content still produces a new blob. - That adds up to \~605 MB of new blobs per release. The push runs at about 1.2 MB/s, which gives the 505 s. - 1.2 MB/s is very slow for a runner in a datacenter, and gzip alone can't account for it. Runner egress or CPU limits are possible causes, but I haven't verified that. Sending fewer bytes fixes the problem either way. - Frequent tags multiply the cost. Pre-release tags (a/b/rc) run the full release too: on 2026-09-16 four tags each pushed \~600 MB. ### 3. Almost half of the biggest layer is never read The `uv sync` layer is 931 MiB uncompressed. 423 MiB of that is uv's download cache, stored at `/home/user/shared/.uv-cache`. - **Why it gets in:** `UV_CACHE_DIR` points to that path, and nothing is mounted there during the kaniko build. - **Why nothing uses it:** at runtime and during site builds, `/home/user/shared` is bind-mounted over that path, so the baked-in copy is hidden. - **Cost:** about 140 MB compressed on every push and every site pull. ### 4. The site update path Upgrading means installing the new RPM or DEB and then running `juice install --rebuild --restart`. The docs say this can take up to 30 minutes. **Pull.** Each site downloads the same \~605 MB of new layers per update. Old base versions are kept, so the devcontainer's podman storage is 33.6 GB, 95% of it reclaimable. **Build.** Juice builds one image per service plus one for singleuser (3 by default, 4 in the devcontainer). These use the same template as the CI base image, so each one repeats this work on top of the new base: - **Two `dnf` runs.** These are not no-ops: the second one upgraded nodejs from 22.23.1 to 22.23.2 in the shared image. Skipping them would change behavior. - **`COPY uv:latest`.** This replaces the base's uv with the site's locally cached `uv:latest`, which can be older. - **`uv sync` of the lab dependencies.** This adds \~570 MB of packages that aren't in the base. All 101 base packages kept their exact versions, apparently because uv uses the base's `uv.lock` as version preferences. - **Singleuser only:** it re-downloads code-server (unpinned; a 973 MB layer) and filebrowser. **Duplicated work.** - Identical environments are each built separately, so the same `uv sync` runs once per environment. That covers task-manager, shared, and any device service on the default environment. - A timestamp in the rendered `pyproject.toml` forces `uv sync` on every build. That's deliberate: a rebuild is supposed to re-resolve. - The JupyterHub image reinstalls juice-core and \~118 MB of dependencies from PyPI on every version bump, with no cache. **Things I checked that are not problems:** - The site's uv cache already persists, because the build unit bind-mounts the shared directory (`environment.py:33`). - `chown -R /env` doesn't copy the venv when the UID matches; that layer is 7 kB. - Bytecode compilation doesn't appear to rewrite the base's `.pyc` files. ### 5. Building the package itself The wheel is 1.6 MB of pure Python with no JS or compile step, and all 258 locked dependencies ship prebuilt wheels. The overhead is in the jobs around the build: - **Package Python** spends \~28 s creating a micromamba env it doesn't need. - **Build docs** spends \~28 s on micromamba just to get Python 3.12, which uv can provide faster. - **RPM and DEB** jobs, on every run: - reinstall ruby and fpm (36 s and 71 s); - pip-install juice from source plus \~100 dependencies, with 0 cache hits (\~60 s); - spend \~61 s each in rpmbuild or deb creation. - **The runner cache is broken.** Every restore and save fails with `403 InvalidAccessKeyId` against DigitalOcean Spaces. It also only caches pip and conda paths, which uv doesn't use. - **Doc build artifacts are huge.** Sphinx's doctrees take 3.3 GB locally (175 files of \~21 MB). This may account for much of the \~140 s Sphinx time; I haven't verified that. - **MR pipelines never build the image**, so template changes are first tested on a release tag. ### 6. Prior work on the branch `origin/perf/optimize-juice-rebuild-times` (unmerged, 2026-09-02) A teammate split the template into a full base template and a slim environment template, added a uv cache mount, and wrote ADRs 0001–0004. It's relevant, but it doesn't fit the non-breaking constraint as it stands: - **It's a breaking change.** ADR-0001 makes the environment build fail when `base_image` is a raw OS image. - **Its speedups don't apply to podman hosts.** - The 289 s → 22 s figure comes from a prebuilt, site-specific base image. - The cache-mount gain was measured with docker BuildKit. Podman hosts already mount a persistent cache. - **It doesn't touch CI:** not the baked-in cache, not layer reuse, and not the render job's dependency on the RPM. - **It agrees on the main lever.** ADR-0003 builds the next image FROM the previous one, which brings pulls down to \~2 MB. ### 7. Levers, ranked by impact against risk (all non-breaking) 1. **Stop baking the uv cache into the base image.** Set `UV_NO_CACHE=1` only for the build (as a build arg), and leave the runtime `UV_CACHE_DIR` unchanged. This saves \~140 MB per push and per pull, and nothing reads the cache anyway. 2. **Add a kaniko registry layer cache** (`--cache`, `--cache-repo`, `--cache-ttl`). The EPEL and dnf layers then keep their digests and are not re-pushed or re-pulled. The trade-off is that OS packages refresh once per cache period instead of every release. Site builds still upgrade the listed packages. - Estimate for 1 and 2 together: push and pull drop from \~605 to \~200 MB per release, and the push from \~505 to \~170 s. 3. **Take the image build off the RPM's critical path.** - Render the build context from the wheel right after Package Python. - Keep today's release gating by listing it explicitly. The release jobs currently wait on the RPM only through the render job. - Drop micromamba from Package Python. - Estimate with 1–3: the tag pipeline drops from \~20 to \~8 min. 4. **Small site-side savings.** - Skip the EPEL reinstall when EPEL is already present. That saves one dnf metadata refresh per environment image. - Add a uv cache mount to the JupyterHub image build. 5. **Bigger follow-ups, each with its own trade-off:** - Reuse the Python layer across releases, so most releases push and pull only \~1–25 MB. Either split dependencies from juice and pin them to a periodic snapshot (older dependency versions in the base), or build FROM the previous image and flatten periodically (layers grow in between). - Build identical environments only once. This needs a rethink of the timestamp and the hook that re-renders at build start. - Prebuild the singleuser tools in CI. - Have fpm install from the wheel, using a prebuilt packaging image. - Fix the runner cache credentials. - Look into the doctree size. - zstd or zstd:chunked would shrink pulls further, but every client must support it, so it isn't safe by default. ## Levers with estimated savings - **Pipeline** figures are measured job timings, except those for levers 7–13, which are estimates. - **`juice install`** figures are all estimates. I haven't timed a real `juice install --rebuild`, so I worked them out from layer sizes and the step timings in the CI logs. They assume a 100 Mbps site link. ### Recommended levers | \# | Lever | Step it affects: time now | `juice install` saving (est.) | Pipeline build + push saving | |----|-------|---------------------------|-------------------------------|------------------------------| | 1 | Stop baking the uv cache into the base image | Site: pulls \~605 MB per update (\~50 s). CI: push takes 505 s | \~15–30 s (140 MB and \~20k fewer files to pull and unpack) | \~130 s per tag | | 2 | Kaniko layer cache, kept in the `ci` repo¹ | Build CI Base Image: 669–777 s (build \~144 s, push \~505 s) | \~30 s (250 MB less), if the OS layer hasn't changed since the site's last update | \~240 s on a cache hit; 0 when the OS layer is refreshed² | | 3 | Render the build context from the wheel and start the image build right after Package Python | Render job: 66–95 s (52 s of it installs the RPM). It also waits for Package RPM (169–226 s) | none (releases publish \~4.5 min sooner) | \~275 s per tag | | 4 | Drop micromamba from Package Python and Build docs | Package Python: 62–85 s. Build docs: 189–290 s. Each spends \~28 s on micromamba | none | \~28 s on every pipeline | | 5 | Skip the EPEL reinstall when EPEL is already present | EPEL step in each of the 3–4 environment images: \~10–15 s each | \~10–15 s with parallel builds; \~40–60 s with sequential builds | none | | 6 | uv cache mount for the JupyterHub image | Hub rebuild re-downloads up to \~118 MB of dependencies on each version bump | \~10 s | none | | | **All of 1–6** | Tag pipeline: \~20 min | \~1–1.5 min | Tag pipeline \~20 → \~8 min on a cache hit, \~12 min when the OS layer is refreshed | ¹ The cache lives in the same registry repository as the image, so a cache miss doesn't upload the layer twice. ² I checked this against the real tag dates since July. About 60% of tags would hit the cache with a 7-day TTL, and about 75% with the kaniko default of 14 days. At 20 Mbps, levers 1, 2 and 6 save \~1 min, \~2 min and \~50 s, so the combined `juice install` saving is \~4 min. Per release, the push and the site pull drop from \~605 MB to \~210 MB on a cache hit, and to \~460 MB when the OS layer is refreshed. ### Follow-ups | \# | Lever | Step it affects: time now | `juice install` saving (est.) | Pipeline build + push saving (est.) | |----|-------|---------------------------|-------------------------------|-------------------------------------| | 7 | Reuse the Python layer across releases | The \~180 MB Python layer is re-pushed and re-pulled every release (size after lever 1) | \~25 s (180 MB less), if dependencies are unchanged | Image job \~200 s shorter; tag pipeline only \~60 s, because DEB becomes the slowest path | | 8 | Build identical environments only once | Each duplicate environment image repeats the dnf steps, a `uv sync` of \~570 MB and the layer commits | \~1–2 min of build work per duplicate³ | none | | 9 | Prebuild the singleuser tools in CI | code-server (973 MB layer) and filebrowser are reinstalled on every update | \~45–90 s | Adds a build and push: \~1 GB once, then cached | | 10 | Build the RPM/DEB from the wheel, in a prebuilt packaging image | Package RPM: 169–226 s. Package DEB: 200–263 s | none | \~70 s per RPM job, \~110 s per DEB job. Main pipeline \~60–80 s shorter, until docs becomes the slowest job | | 11 | Fix the runner cache credentials | Every cache restore and save fails with a 403 | none | A few seconds per job until a uv cache is added | | 12 | Investigate the doctree size | Build docs: 189–290 s (\~138 s in Sphinx, 3.3 GB of doctrees) | none | Unknown until profiled | | 13 | zstd compression (not safe by default: older clients can't pull it) | Layers are gzip today | \~10% fewer bytes, and faster unpacking | \~10% less to push | ³ A default install has one duplicate (task-manager and shared share an environment); the devcontainer has two. With parallel builds, the wall-clock saving is smaller.