FIPS Runner panics on TLS 1.3 handshake: "tls: HKDF-Expand-Label invocation failed unexpectedly"
Generated with the assistance of GitLab Duo
Summary
GitLab Runner FIPS binaries panic on TLS 1.3 handshakes with:
panic: tls: HKDF-Expand-Label invocation failed unexpectedly
goroutine 55 [running]:
crypto/tls.expandLabel(...)
/usr/local/go/src/crypto/tls/key_schedule.go:294 +0x319
crypto/tls.(*cipherSuiteTLS13).trafficKey(...)
/usr/local/go/src/crypto/tls/key_schedule.go:33 +0xfa
crypto/tls.(*halfConn).setTrafficSecret(...)
/usr/local/go/src/crypto/tls/conn.go:233 +0x67
crypto/tls.(*Conn).setWriteTrafficSecret(...)
/usr/local/go/src/crypto/tls/conn.go:1706
crypto/tls.(*clientHandshakeStateTLS13).establishHandshakeKeys(...)
/usr/local/go/src/crypto/tls/handshake_client_tls13.go:498 +0x1c5
...Confirmed on v19.0.3, v19.1.2 and v19.2.2, both reproduced during Packer/AMI builds on FIPS-enabled hosts:
time="2026-09-01T05:00:01Z" level=info msg="FIPS mode is enabled. Using an external SSL library."
Runtime platform arch=amd64 os=linux pid=5363 revision=e951e492 version=19.0.3
panic: tls: HKDF-Expand-Label invocation failed unexpectedlytime="2026-09-01T04:28:18Z" level=info msg="FIPS mode is enabled. Using an external SSL library."
Runtime platform arch=amd64 os=linux pid=5268 revision=f448231c version=19.1.2
panic: tls: HKDF-Expand-Label invocation failed unexpectedlytime="2026-08-24T07:00:40Z" level=info msg="FIPS mode is enabled. Using an external SSL library."
Runtime platform arch=amd64 os=linux pid=5357 revision=343288f1 version=19.2.2
panic: tls: HKDF-Expand-Label invocation failed unexpectedlyRoot cause (known upstream)
This is the same defect described in the golang-fips patch *-hkdf-expandlabel-read.patch (first shipped in go1.25.9-2-openssl-fips), which replaces the io.ReadAll-based HKDF-Expand-Label read in crypto/tls.expandLabel() with a single r.Read, converting a failure of the underlying OpenSSL EVP_PKEY_derive call into a fatal panic instead of a normal error. Background/incident writeups:
- Gitaly hit the exact same panic on FIPS+TLS builds:
gitlab-com/gl-infra/gitlab-dedicated/team#13088and the incident reviewgitlab-com/gl-infra/gitlab-dedicated/team#13590. - The fix for the Omnibus/Gitaly build chain removed the patch from the FIPS Go toolchain build: gitlab-omnibus-builder!491 (merged) ("Skip golang-fips hkdf-expandlabel-read patch in FIPS Go build"), backported via omnibus-gitlab!9600 (merged), !9602, !9603, !9604.
That fix does not apply to gitlab-runner. gitlab-runner builds its FIPS binaries with a separate toolchain pipeline:
- The
binariesCI job (.gitlab/ci/build.gitlab-ci.yml) runsmake ... runner-bin-fipsinside the${RUNNER_IMAGES_REGISTRY}/ubi-go:${RUNNER_IMAGES_VERSION}image, whereRUNNER_IMAGES_REGISTRY = registry.gitlab.com/gitlab-org/ci-cd/runner-tools/base-images(see.gitlab/ci/_common.gitlab-ci.yml). - That
ubi-goimage is built in gitlab-org/ci-cd/runner-tools/base-images fromdockerfiles/ci/Dockerfile.go, which clonesgolang-fips/goat branchgo${GOLANG_VERSION}-${GO_FIPS_VERSION_SUFFIX}and runs the upstreamfull-initialize-repo.shpatch step. dockerfiles/ci/docker-bake.hcl(main) currently pinsGOLANG_VERSION = "1.26.5"andGO_FIPS_VERSION_SUFFIX = "1-openssl-fips"(branchgo1.26.5-1-openssl-fips).Dockerfile.goonly removespatches/001-fix-linkage.patch. It does not remove the hkdf-expandlabel-read patch.- Note that
Dockerfile.goclonesgolang-fips/goby branch name, not a pinned commit SHA, so the toolchain content of a givenubi-goimage tag cannot be inferred from base-images git history alone — it depends on the state of the branch at the time the image was built.
I could not find any MR or issue in gitlab-org/ci-cd/runner-tools/base-images or gitlab-org/gitlab-runner addressing this (searched for "hkdf", "HKDF", "expandlabel" in both projects — no results). Whether go1.26.5-1-openssl-fips still ships the patch under a renumbered name needs to be confirmed directly against the golang-fips/go patches for that tag, but go1.26.4 is documented to carry it as 005-hkdf-expandlabel-read.patch (per the omnibus-builder backport MR description), and no removal has landed in either repo since.
Impact
Any GitLab Runner FIPS binary/image built from a golang-fips/go branch carrying the patch will panic on the first TLS 1.3 handshake it performs as a client (e.g. connecting to a GitLab instance over TLS 1.3), crashing the runner process.
The v19.0.3 reproduction proves the defective patch is present in the ubi-go:0.0.43 toolchain (branch go1.26.3-1-openssl-fips), which extends the affected range beyond 19.2.x:
| Runner tags | base-images (RUNNER_IMAGES_VERSION) |
golang-fips branch | Status |
|---|---|---|---|
| v19.0.0–v19.0.3, v19.1.0–v19.1.2 | 0.0.43 | go1.26.3-1-openssl-fips |
Affected — panic reproduced on v19.0.3 (multiple runs); all these tags are built with the same ubi-go:0.0.43 toolchain |
| v19.2.0–v19.2.3 | 0.0.47 | go1.26.4-1-openssl-fips |
Affected — panic reproduced on v19.2.2; go1.26.4 is documented to carry the patch |
| v19.3.0–v19.3.1 | 0.0.48 | go1.26.5-1-openssl-fips |
Not yet confirmed via a reproduction — but built from a later branch of the same line, and no patch removal has landed in either repo |
(Toolchain mapping verified from .gitlab/ci/_common.gitlab-ci.yml at each runner tag and dockerfiles/ci/docker-bake.hcl at each base-images tag. Tag ranges are complete as of 2026-09-01.)
Given the confirmed panics on both v19.0.3 and v19.2.2, no 19.x FIPS binary is currently known to be safe. For reference, the last pre-19.0 releases (v18.11.3/v18.11.4, base-images 0.0.40) were built from branch go1.25.7-1-openssl-fips, which predates go1.25.9-2-openssl-fips where the patch first shipped; however, no 18.11.x binary has been tested against this defect yet.
Suggested fix
Mirror gitlab-omnibus-builder!491 (merged) in gitlab-org/ci-cd/runner-tools/base-images: delete the *-hkdf-expandlabel-read.patch file from the golang-fips checkout in dockerfiles/ci/Dockerfile.go before full-initialize-repo.sh applies the patch set, e.g.:
RUN rm /tmp/go/patches/*-hkdf-expandlabel-read.patchplaced alongside the existing removal of 001-fix-linkage.patch.
Additionally, consider pinning the golang-fips/go clone in Dockerfile.go to a commit SHA instead of a branch name, so the toolchain content of each ubi-go image tag is reproducible and auditable.
Once a corrected base image is available, affected runner releases need FIPS binaries/images rebuilt against it.