Fix logrotate usage writer mutating record labels
What does this merge request do?
The logrotate usage writer currently adds its configured labels directly to record.Labels. A usage_log.Record is passed by value, but its labels are a map, so this changes the caller's record as well.
This MR copies the labels before applying the writer's labels. The JSON written by logrotate stays the same: record labels are preserved, and writer labels still win when both maps contain the same key.
Why was this needed?
UsageLogRecordFrom() gets its labels from RunnerSettings.ComputedLabels(), which returns the runner's backing map. The same map is also used in job update requests.
This creates a race when a job finishes:
- the trace watcher can be encoding
UpdateJobRequest.Info.Labelsinjson.Marshal(); - at the same time,
finishBuild()can pass the shared map to the logrotate writer, which writes its configured labels into it.
We saw this terminate a Runner 19.2.2 manager with fatal error: concurrent map iteration and map write. The stack ended in json.Marshal -> doJSON -> UpdateJob -> clientJobTrace.touchJob, which is the reader side of this race.
The last production log lines put the two operations 354 microseconds apart:
time="2026-08-27T11:11:41.980348027Z" level=info msg="Storing usage log information"
time="2026-08-27T11:11:41.980702463Z" level=info msg="Updating job..."
fatal error: concurrent map iteration and map writeRunner emits the first line immediately before usageLogger.Store(record) and the second before doJSON() marshals the update request. The panic stack shows that marshal running in the trace watcher. The dump uses the default GOTRACEBACK=single, so it does not include the writer goroutine. The close timing is consistent with the race, but the log order alone does not prove the overlap.
The bug also has a deterministic effect. After the first job, logrotate labels were left in the runner's shared map. Later job requests and updates then sent those labels to GitLab as Info.Labels.
The copy belongs in the writer rather than only in UsageLogRecordFrom(). The writer is the code changing caller-owned data. Fixing it there also covers other record producers and stops one MultiWriter destination from changing what the next destination receives.
Closes #39723 (closed)
Testing
The tests check that:
Store()does not change the original record's labels;- the real
RunnerConfig -> ComputeLabels -> UsageLogRecordFrom -> Storepath does not change the runner's computed labels; - the stored JSON contains the merged labels, with writer labels taking precedence.
The test fails with the previous implementation and passes with this change.
Validation run locally:
go test -race ./helpers/usage_log/...
go test ./common ./network
go build ./...
make lintThe changed usage-log packages also passed during go test -race ./.... The full run had unrelated failures in an architecture-sensitive Kubernetes test and a commands/helpers wait-delay test on the local arm64 environment. go vet ./... reported existing generated-mock signature warnings; go vet ./helpers/usage_log/... passed.