## Overview Extend LabKit to provide an instrumented object store client (S3-compatible) that wraps blob storage operations with observability primitives out of the box. ## Motivation Artifact Registry stores blobs in object storage (GCS/S3). Without a shared instrumented client, teams must wire up tracing, metrics, and logging independently, resulting in gaps in observability and inconsistent telemetry across services. ## Goals - Provide a `labkit/v2/objectstore` package with an instrumented S3-compatible client - Automatically inject **distributed tracing** spans (OpenTelemetry) for every object store operation - Emit **Prometheus metrics** for operation latency, error rates, and byte throughput - Enrich **structured logs** with operation name, bucket, object key (sanitised), duration, and error details - Support pluggable backends (AWS S3, GCS via S3 compatibility, MinIO) - Expose a clean interface that wraps the AWS SDK v2 S3 client (or equivalent) without leaking internals ## Acceptance Criteria - [ ] `objectstore.NewClient(cfg)` returns an instrumented client compatible with the AWS SDK v2 S3 interface - [ ] Each operation (`GetObject`, `PutObject`, `DeleteObject`, `HeadObject`, `ListObjectsV2`) produces an OpenTelemetry span with standard semantic conventions (`rpc.system`, `rpc.service`, `rpc.method`, `server.address`, etc.) - [ ] Prometheus histogram `labkit_objectstore_operation_duration_seconds` is registered and populated, labelled by operation and bucket - [ ] Errors are counted via `labkit_objectstore_operation_errors_total` labelled by operation and error code - [ ] Byte throughput is tracked via `labkit_objectstore_bytes_transferred_total` labelled by direction (`upload`/`download`) - [ ] Structured log entries include `objectstore_operation`, `objectstore_bucket`, `objectstore_duration_ms`, and `error` fields when applicable - [ ] Unit tests cover tracing, metrics, and logging for `GetObject`, `PutObject`, and `DeleteObject` - [ ] Integration test or example demonstrates usage against a local MinIO instance ## References - Related epic: [&376](https://gitlab.com/groups/gitlab-org/quality/-/work_items/376) - Existing pattern: [labkit/v2/postgres: Add marginalia SQL comment injection](https://gitlab.com/gitlab-org/quality/quality-engineering/team-tasks/-/work_items/4383) - OpenTelemetry semantic conventions for object stores: https://opentelemetry.io/docs/specs/semconv/object-stores/s3/