perf: instrument GCS chunk upload timing for blob PUT diagnostics
What does this MR do?
Adds timing instrumentation to the blob upload path to help diagnose the intermittent 10–16 minute stall on a single blob PUT /blobs/uploads reported in https://gitlab.com/gitlab-com/request-for-help/-/work_items/4993.
Registry logs for the slow upload show the entire stall is in copyFullPayload() — the body-receive-to-GCS streaming phase. Everything after (GCS finalize, server-side copy, digest validation, DB write) completes in ~1.3 seconds. We have no visibility into the ~17 putChunk() calls that happen during this phase, so we can't determine whether GCS is the bottleneck or something else.
Changes
1. duration_s on "payload copied" (registry/handlers/helpers.go)
Adds explicit wall-clock duration to the existing "payload copied" Info log, removing the need to compute timing from timestamp gaps between separate log entries.
2. Chunk timing on the GCS writer (registry/storage/driver/gcs/gcs.go)
Adds three fields to the writer struct that accumulate across all putChunk() calls — both from writeChunk() during body streaming (~16 calls for a 90MB blob) and the final flush in Commit(). Reports once at Commit() as "resumable upload committed":
| Field | Meaning |
|---|---|
chunk_attempts |
Total putChunk() calls across the upload lifetime |
chunk_total_s |
Cumulative wall time inside putChunk() |
chunk_max_s |
Slowest single putChunk() call |
Diagnostic logic: comparing chunk_total_s against "payload copied".duration_s tells us where the time went:
chunk_total_s ≈ duration_s→ GCS was the bottleneckchunk_total_s << duration_s→ stall is outsideputChunk(), need to look elsewhere
What is not changed
No behaviour changes. All additions are purely observational — clock reads and one log entry per resumable upload.
Related to https://gitlab.com/gitlab-com/request-for-help/-/work_items/4993
Author checklist
- CODEOWNERS Review: This MR requires approval from at least one CODEOWNER per category/file.
-
perf: Focuses on performance improvements that don't introduce new features or fix bugs, triggers a patch release. -
MR contains database changes— N/A -
Change contains a breaking change— N/A -
Change is considered high risk— N/A -
Changes cannot be rolled back— N/A