ci: Build each Linux server target on a runner of its architecture
What does this MR do and why?
Implements the approach agreed in #53 (proposed by @vtak, agreed by @ealcantara): build each Linux server variant natively on a runner of its target architecture, so the packaged native modules actually match the package name.
Problem
Every vscode-reh-web-linux-arm64-*.tar.gz published so far ships x86-64 native binaries. Checked on the latest release (1.94.0-1.0.0-dev-20251209191616):
| file in the linux-arm64 tarball | ELF machine |
|---|---|
./node |
aarch64 |
node_modules/node-pty/build/Release/pty.node |
x86-64 |
node_modules/@vscode/spdlog/build/Release/spdlog.node |
x86-64 |
node_modules/@parcel/watcher/build/Release/watcher.node |
x86-64 |
node_modules/kerberos/build/Release/kerberos.node |
x86-64 |
node_modules/native-watchdog/build/Release/watchdog.node |
x86-64 |
node_modules/@vscode/deviceid/build/Release/windows.node |
x86-64 |
node_modules/@vscode/ripgrep/bin/rg |
x86-64 |
build-server runs one npm ci on an amd64 runner, then loops all five targets through the vscode-reh-web-<os>-<arch>-min-ci gulp tasks. Those are packaging tasks: upstream's build/gulpfile.reh.js copies remote/node_modules verbatim from the host and only downloads the node runtime per target - which is why the arm64 "build" takes 12 seconds and only its node binary is correct. Upstream doesn't support cross-compiling the native modules; Microsoft's own pipelines run one job per architecture with npm_config_arch set (build/azure-pipelines/linux/product-build-linux.yml).
Changes
- Split
build-serverintobuild-server-amd64(saas-linux-medium-amd64:darwin-x64 darwin-arm64 win32-x64 linux-x64) andbuild-server-arm64(saas-linux-medium-arm64:linux-arm64), sharing a.build-server-basetemplate. Both emit into.build/vscode-reh-web, sopublish-packagesand the released file set are unchanged. scripts/build-and-bundle-server.shreads the target list from$VSCODE_BUILD_TARGETS(default: the previous hardcoded list, for local builds).- The dependency cache key is sharded per architecture (
vscode-fork-dependency-cache-{amd64,arm64}): the cache contains compiled native modules, so an arm64 job must never restore amd64node_modules.push-cachekeeps pushing only the amd64 cache for now, so the arm64 job installs cold (~10 extra minutes); an arm64 cache twin can be a follow-up. - New
scripts/verify-server-package-arch.shruns after every server build and fails the job if any packaged native binary (*.node, ripgrep, thenoderuntime) doesn't match the architecture of the host-matching Linux target. This bug has shipped silently since 2022 (#8 (closed), #12 (closed), #53); with the guard it cannot ship silently again. (It reads the ELFe_machinebytes withod, sincefileisn't installed in the CI image.)
Wall-clock should stay roughly flat: the two jobs run in parallel and each builds fewer targets than before.
Dependencies
- Depends on gitlab-build-images!1111 (merged): this project's CI image (
gitlab-vscode-build) is amd64-only today, so the arm64 job can't start until the image exists for arm64. Draft until that merges. - Releases are maintainer-only tag pipelines, so the first fixed
linux-arm64package ships with the next tag cut after this merges.
Open questions for maintainers
- The
darwin-*/win32-x64packages have never contained working native modules either (same root cause - they get this host's linux-x64 binaries). This MR deliberately keeps packaging them unchanged on the amd64 job to avoid changing the published artifact set. Is anything consuming them, or should a follow-up drop them? - @ealcantara - the
split-web-ide-and-workspaces-forksbranch touches the same files; happy to rebase this MR if that lands first.
Follow-ups (tracked in gitlab#550128)
Per the sequencing in gitlab-build-images!905 (merged)#note_2566824178: next comes making the workspaces-tools image fetch per-arch fork tarballs (fetch_assets.sh currently downloads the first asset URL it finds) and building it multi-arch, then the default workspace image, then switching the Rails digest pins back to tags. We are happy to contribute those MRs as well.
Who's asking
We run GitLab Workspaces on an AKS cluster whose node pools are arm64-by-default, and currently maintain a dedicated amd64 node pool only because of this issue. The demand predates us: gitlab#392693 has carried customer interest since 2023, and gitlab#550128 marks the work open for community contribution. Related prior contribution: gitlab-org/workspaces/gitlab-workspaces-proxy!52 (merged) (multi-arch proxy image).
Testing attestation (2026-09-04)
Artifacts under test come from the first fully green pipeline, https://gitlab.com/gitlab-org/gitlab-web-ide-vscode-fork/-/pipelines/2820298281 (build-server-arm64 on saas-linux-medium-arm64, 67 min).
1. In-job arch guard (scripts/verify-server-package-arch.sh, job 16307470784):
OK: all 9 ELF binaries in vscode-reh-web-linux-arm64 are linux-arm64 (e_machine 0xb700); 1 non-ELF file(s) skipped
2. Independent ELF audit of the packaged vscode-reh-web-linux-arm64-1.94.0-1.0.0-dev-20260904124619.tar.gz (file(1) on an unrelated machine). Every binary that shipped as x86-64 in previous releases is now aarch64:
| binary | result |
|---|---|
node |
ELF aarch64 |
node_modules/node-pty/build/Release/pty.node |
ELF aarch64 |
node_modules/@vscode/spdlog/build/Release/spdlog.node |
ELF aarch64 |
node_modules/@parcel/watcher/build/Release/watcher.node |
ELF aarch64 |
node_modules/kerberos/build/Release/kerberos.node |
ELF aarch64 |
node_modules/native-watchdog/build/Release/watchdog.node |
ELF aarch64 |
node_modules/@vscode/deviceid/build/Release/windows.node |
ELF aarch64 |
node_modules/@vscode/ripgrep/bin/rg |
ELF aarch64 |
extensions/ms-vscode.js-debug/src/w32appcontainertokens-*.node |
PE32+ x86-64 — Windows-only helper that upstream ships on every platform |
Highest GLIBC_ symbol referenced by the shipped binaries: 2.28 — the documented workspace container floor is preserved.
3. Runtime test on real hardware, both architectures. Containers running Debian 10 (buster) — i.e. glibc 2.28, the minimum documented workspace platform — on AKS: the linux-arm64 tarball on an arm64 (Ampere) node, and the linux-x64 tarball from the same pipeline on an amd64 node. (libgssapi-krb5-2 installed first; workspace images ship it, bare debian:10 does not.) Results identical on aarch64 and x86_64:
./node -v→v20.16.0require()ofnode-pty,@vscode/spdlog,kerberos,native-watchdog,@parcel/watcher→ all loadnode_modules/@vscode/ripgrep/bin/rg --version→ripgrep 13.0.0bin/gitlab-webide-server --host 127.0.0.1 --port 8000 --without-connection-token→Extension host agent listening on 8000;GET /version→ HTTP 200, bodyd78a74bcdfad14d5d3b1b782f87255d802b57511