feat(v2/redis): add instrumented Redis client

Luke Hollindarequested to merge
lh-initial-redis-client into master

Parent Issue: gitlab-org/quality/quality-engineering/team-tasks#4455 (closed)

Preamble:

This implementation grew to be larger than I had initially expected. Apologies to my reviewers; I plan on iterating with smaller MRs in the future.

I assume that this will require some back and forth to flesh out the exact requirements. I hope this is an acceptable first step.

Summary:

Introduces gitlab.com/gitlab-org/labkit/v2/redis, an instrumented Redis client that satisfies app.Component and supports single-node, Sentinel, and Cluster topologies through a single unified Config. Instrumentation (OTel tracing, Prometheus metrics, structured logging) is applied as hooks at construction time and requires no changes at call sites.

The package wraps https://github.com/redis/go-redis and exposes goredis.UniversalClient directly via Client.Redis(), giving callers full access to the go-redis API without any adapter layer.

Key Things to Look Out For:

  • 98% of my manual testing effort has been focused on the single node Redis Client. Thought there are integration tests for Sentinal + Cluster clients.
  • I've set up a local in-process Redis stub for use in unit testing v2/testing/redistest/redistest.go)
  • goredis.UniversalOptions is a very large struct. We can either expose this directly during client initialization or wrap the required fields. I've elected to wrap a subset of the options (19 of 49) that I expected to be most commonly used or are required to differentiate which client type should be created. Would love some feedback on if this is a good approach or not.
  • v2/redis/integration_test.go includes instructions and integration tests to be run alongside Redis (available as Docker Compose services in v2/redis/testdata/). I'll hook these up in CI as a follow up action.
  • Metrics are registered in v2/redis/pool_collector.go and v2/redis/metrics_hook.go. These live in separate files because the pool metrics are pulled and are not part of the hook, which is responsible only for the per-command metrics.

Prometheus Metrics

When a prometheus.Registerer is provided, individual command and pool metrics are automatically registered.

Metric Type Labels Description
gitlab_redis_command_duration_seconds Histogram name, command Duration of each Redis command. command="pipeline" for pipelines. Uses prometheus.DefBuckets.
gitlab_redis_command_errors_total Counter name, command Commands that returned an error. redis.Nil is excluded.
gitlab_redis_pool_hit_total Counter name Pool connections found without waiting.
gitlab_redis_pool_miss_total Counter name Pool misses — a new connection was established.
gitlab_redis_pool_timeout_total Counter name Requests that timed out waiting for a pool connection.
gitlab_redis_pool_conn_total_current Gauge name Current total connections in the pool.
gitlab_redis_pool_conn_idle_current Gauge name Current idle connections in the pool.
gitlab_redis_pool_conn_stale_total Counter name Connections closed because they exceeded idle/lifetime limits.

Example Usage:

Note

Start a local Redis server before running the example usage: docker-compose -f ./v2/redis/testdata/docker-compose.yml up -d

func main() {
	ctx := context.Background()

	// METRICS
	m, _ := metrics.New()

	// LOGGER
	logger := log.New()

	// REDIS CLIENT
	client, err := redis.NewWithConfig(&redis.Config{
		Addrs:      []string{"localhost:6379"}, # Locally running Redis instance
		Name:       "lhollinda-instance",
		Registerer: m.Registerer(),
		Logger: logger,
	})
	if err != nil {
		panic(err)
	}

	_ = client.Start(ctx)
	defer client.Shutdown(ctx)

	// DEMO REDIS COMMANDS
	_, _ = client.Redis().SetEx(ctx, "testKey", "testValue", 100*time.Second).Result()
	_, _ = client.Redis().GetEx(ctx, "testKey", 1*time.Second).Result()

	// SERVE METRICS
	http.Handle("/metrics", m.Handler())
}
Resulting logs:
  • With some manually added comments
  • Started with GITLAB_LOG_LEVEL=debug go run main.go
### go-redis connecting to the server
{"time":"2026-05-20T21:47:49.122718Z","level":"DEBUG","msg":"redis command","name":"lhollinda-instance","redis_command":"hello","redis_duration_ms":0.970417,"redis_key":"3"}
{"time":"2026-05-20T21:47:49.123385Z","level":"DEBUG","msg":"redis pipeline","name":"lhollinda-instance","redis_command":"pipeline","redis_pipeline_size":1,"redis_duration_ms":0.5575000000000001}
{"time":"2026-05-20T21:47:49.123893Z","level":"WARN","msg":"redis command","name":"lhollinda-instance","redis_command":"client","redis_duration_ms":0.492667,"redis_key":"maint_notifications","error_message":"ERR unknown subcommand 'maint_notifications'. Try CLIENT HELP."}
{"time":"2026-05-20T21:47:49.124401Z","level":"DEBUG","msg":"redis pipeline","name":"lhollinda-instance","redis_command":"pipeline","redis_pipeline_size":2,"redis_duration_ms":0.481459}
### Ping Command from Client.Start()
{"time":"2026-05-20T21:47:49.124886Z","level":"DEBUG","msg":"redis command","name":"lhollinda-instance","redis_command":"ping","redis_duration_ms":4.4850840000000005}
{"time":"2026-05-20T21:47:49.124896Z","level":"INFO","msg":"redis client connected","name":"lhollinda-instance"}
### Set + Get Commands
{"time":"2026-05-20T21:47:49.125442Z","level":"DEBUG","msg":"redis command","name":"lhollinda-instance","redis_command":"setex","redis_duration_ms":0.49474999999999997,"redis_key":"testKey"}
{"time":"2026-05-20T21:47:49.126005Z","level":"DEBUG","msg":"redis command","name":"lhollinda-instance","redis_command":"getex","redis_duration_ms":0.518208,"redis_key":"testKey"}
Resulting metrics:
> curl localhost:2112/metrics
# HELP gitlab_redis_command_duration_seconds Duration of Redis commands in seconds.
# TYPE gitlab_redis_command_duration_seconds histogram
gitlab_redis_command_duration_seconds_bucket{command="client",name="lhollinda-instance",le="0.005"} 1
gitlab_redis_command_duration_seconds_bucket{command="client",name="lhollinda-instance",le="0.01"} 1
gitlab_redis_command_duration_seconds_bucket{command="client",name="lhollinda-instance",le="0.025"} 1
gitlab_redis_command_duration_seconds_bucket{command="client",name="lhollinda-instance",le="0.05"} 1
gitlab_redis_command_duration_seconds_bucket{command="client",name="lhollinda-instance",le="0.1"} 1
gitlab_redis_command_duration_seconds_bucket{command="client",name="lhollinda-instance",le="0.25"} 1
gitlab_redis_command_duration_seconds_bucket{command="client",name="lhollinda-instance",le="0.5"} 1
gitlab_redis_command_duration_seconds_bucket{command="client",name="lhollinda-instance",le="1"} 1
gitlab_redis_command_duration_seconds_bucket{command="client",name="lhollinda-instance",le="2.5"} 1
gitlab_redis_command_duration_seconds_bucket{command="client",name="lhollinda-instance",le="5"} 1
gitlab_redis_command_duration_seconds_bucket{command="client",name="lhollinda-instance",le="10"} 1
gitlab_redis_command_duration_seconds_bucket{command="client",name="lhollinda-instance",le="+Inf"} 1
gitlab_redis_command_duration_seconds_sum{command="client",name="lhollinda-instance"} 0.000335625
gitlab_redis_command_duration_seconds_count{command="client",name="lhollinda-instance"} 1
gitlab_redis_command_duration_seconds_bucket{command="getex",name="lhollinda-instance",le="0.005"} 1
gitlab_redis_command_duration_seconds_bucket{command="getex",name="lhollinda-instance",le="0.01"} 1
gitlab_redis_command_duration_seconds_bucket{command="getex",name="lhollinda-instance",le="0.025"} 1
gitlab_redis_command_duration_seconds_bucket{command="getex",name="lhollinda-instance",le="0.05"} 1
gitlab_redis_command_duration_seconds_bucket{command="getex",name="lhollinda-instance",le="0.1"} 1
gitlab_redis_command_duration_seconds_bucket{command="getex",name="lhollinda-instance",le="0.25"} 1
gitlab_redis_command_duration_seconds_bucket{command="getex",name="lhollinda-instance",le="0.5"} 1
gitlab_redis_command_duration_seconds_bucket{command="getex",name="lhollinda-instance",le="1"} 1
gitlab_redis_command_duration_seconds_bucket{command="getex",name="lhollinda-instance",le="2.5"} 1
gitlab_redis_command_duration_seconds_bucket{command="getex",name="lhollinda-instance",le="5"} 1
gitlab_redis_command_duration_seconds_bucket{command="getex",name="lhollinda-instance",le="10"} 1
gitlab_redis_command_duration_seconds_bucket{command="getex",name="lhollinda-instance",le="+Inf"} 1
gitlab_redis_command_duration_seconds_sum{command="getex",name="lhollinda-instance"} 0.000330458
gitlab_redis_command_duration_seconds_count{command="getex",name="lhollinda-instance"} 1
gitlab_redis_command_duration_seconds_bucket{command="hello",name="lhollinda-instance",le="0.005"} 1
gitlab_redis_command_duration_seconds_bucket{command="hello",name="lhollinda-instance",le="0.01"} 1
gitlab_redis_command_duration_seconds_bucket{command="hello",name="lhollinda-instance",le="0.025"} 1
gitlab_redis_command_duration_seconds_bucket{command="hello",name="lhollinda-instance",le="0.05"} 1
gitlab_redis_command_duration_seconds_bucket{command="hello",name="lhollinda-instance",le="0.1"} 1
gitlab_redis_command_duration_seconds_bucket{command="hello",name="lhollinda-instance",le="0.25"} 1
gitlab_redis_command_duration_seconds_bucket{command="hello",name="lhollinda-instance",le="0.5"} 1
gitlab_redis_command_duration_seconds_bucket{command="hello",name="lhollinda-instance",le="1"} 1
gitlab_redis_command_duration_seconds_bucket{command="hello",name="lhollinda-instance",le="2.5"} 1
gitlab_redis_command_duration_seconds_bucket{command="hello",name="lhollinda-instance",le="5"} 1
gitlab_redis_command_duration_seconds_bucket{command="hello",name="lhollinda-instance",le="10"} 1
gitlab_redis_command_duration_seconds_bucket{command="hello",name="lhollinda-instance",le="+Inf"} 1
gitlab_redis_command_duration_seconds_sum{command="hello",name="lhollinda-instance"} 0.000769333
gitlab_redis_command_duration_seconds_count{command="hello",name="lhollinda-instance"} 1
gitlab_redis_command_duration_seconds_bucket{command="ping",name="lhollinda-instance",le="0.005"} 1
gitlab_redis_command_duration_seconds_bucket{command="ping",name="lhollinda-instance",le="0.01"} 1
gitlab_redis_command_duration_seconds_bucket{command="ping",name="lhollinda-instance",le="0.025"} 1
gitlab_redis_command_duration_seconds_bucket{command="ping",name="lhollinda-instance",le="0.05"} 1
gitlab_redis_command_duration_seconds_bucket{command="ping",name="lhollinda-instance",le="0.1"} 1
gitlab_redis_command_duration_seconds_bucket{command="ping",name="lhollinda-instance",le="0.25"} 1
gitlab_redis_command_duration_seconds_bucket{command="ping",name="lhollinda-instance",le="0.5"} 1
gitlab_redis_command_duration_seconds_bucket{command="ping",name="lhollinda-instance",le="1"} 1
gitlab_redis_command_duration_seconds_bucket{command="ping",name="lhollinda-instance",le="2.5"} 1
gitlab_redis_command_duration_seconds_bucket{command="ping",name="lhollinda-instance",le="5"} 1
gitlab_redis_command_duration_seconds_bucket{command="ping",name="lhollinda-instance",le="10"} 1
gitlab_redis_command_duration_seconds_bucket{command="ping",name="lhollinda-instance",le="+Inf"} 1
gitlab_redis_command_duration_seconds_sum{command="ping",name="lhollinda-instance"} 0.002563625
gitlab_redis_command_duration_seconds_count{command="ping",name="lhollinda-instance"} 1
gitlab_redis_command_duration_seconds_bucket{command="pipeline",name="lhollinda-instance",le="0.005"} 2
gitlab_redis_command_duration_seconds_bucket{command="pipeline",name="lhollinda-instance",le="0.01"} 2
gitlab_redis_command_duration_seconds_bucket{command="pipeline",name="lhollinda-instance",le="0.025"} 2
gitlab_redis_command_duration_seconds_bucket{command="pipeline",name="lhollinda-instance",le="0.05"} 2
gitlab_redis_command_duration_seconds_bucket{command="pipeline",name="lhollinda-instance",le="0.1"} 2
gitlab_redis_command_duration_seconds_bucket{command="pipeline",name="lhollinda-instance",le="0.25"} 2
gitlab_redis_command_duration_seconds_bucket{command="pipeline",name="lhollinda-instance",le="0.5"} 2
gitlab_redis_command_duration_seconds_bucket{command="pipeline",name="lhollinda-instance",le="1"} 2
gitlab_redis_command_duration_seconds_bucket{command="pipeline",name="lhollinda-instance",le="2.5"} 2
gitlab_redis_command_duration_seconds_bucket{command="pipeline",name="lhollinda-instance",le="5"} 2
gitlab_redis_command_duration_seconds_bucket{command="pipeline",name="lhollinda-instance",le="10"} 2
gitlab_redis_command_duration_seconds_bucket{command="pipeline",name="lhollinda-instance",le="+Inf"} 2
gitlab_redis_command_duration_seconds_sum{command="pipeline",name="lhollinda-instance"} 0.0006498750000000001
gitlab_redis_command_duration_seconds_count{command="pipeline",name="lhollinda-instance"} 2
gitlab_redis_command_duration_seconds_bucket{command="setex",name="lhollinda-instance",le="0.005"} 1
gitlab_redis_command_duration_seconds_bucket{command="setex",name="lhollinda-instance",le="0.01"} 1
gitlab_redis_command_duration_seconds_bucket{command="setex",name="lhollinda-instance",le="0.025"} 1
gitlab_redis_command_duration_seconds_bucket{command="setex",name="lhollinda-instance",le="0.05"} 1
gitlab_redis_command_duration_seconds_bucket{command="setex",name="lhollinda-instance",le="0.1"} 1
gitlab_redis_command_duration_seconds_bucket{command="setex",name="lhollinda-instance",le="0.25"} 1
gitlab_redis_command_duration_seconds_bucket{command="setex",name="lhollinda-instance",le="0.5"} 1
gitlab_redis_command_duration_seconds_bucket{command="setex",name="lhollinda-instance",le="1"} 1
gitlab_redis_command_duration_seconds_bucket{command="setex",name="lhollinda-instance",le="2.5"} 1
gitlab_redis_command_duration_seconds_bucket{command="setex",name="lhollinda-instance",le="5"} 1
gitlab_redis_command_duration_seconds_bucket{command="setex",name="lhollinda-instance",le="10"} 1
gitlab_redis_command_duration_seconds_bucket{command="setex",name="lhollinda-instance",le="+Inf"} 1
gitlab_redis_command_duration_seconds_sum{command="setex",name="lhollinda-instance"} 0.000307583
gitlab_redis_command_duration_seconds_count{command="setex",name="lhollinda-instance"} 1
# HELP gitlab_redis_command_errors_total Total number of Redis command errors, excluding redis.Nil.
# TYPE gitlab_redis_command_errors_total counter
gitlab_redis_command_errors_total{command="client",name="lhollinda-instance"} 1
# HELP gitlab_redis_pool_conn_idle_current Current number of idle connections in the pool.
# TYPE gitlab_redis_pool_conn_idle_current gauge
gitlab_redis_pool_conn_idle_current{name="lhollinda-instance"} 1
# HELP gitlab_redis_pool_conn_stale_total Number of connections removed from the pool because they were stale.
# TYPE gitlab_redis_pool_conn_stale_total counter
gitlab_redis_pool_conn_stale_total{name="lhollinda-instance"} 0
# HELP gitlab_redis_pool_conn_total_current Current number of connections in the pool.
# TYPE gitlab_redis_pool_conn_total_current gauge
gitlab_redis_pool_conn_total_current{name="lhollinda-instance"} 1
# HELP gitlab_redis_pool_hit_total Number of times a connection was found in the pool.
# TYPE gitlab_redis_pool_hit_total counter
gitlab_redis_pool_hit_total{name="lhollinda-instance"} 2
# HELP gitlab_redis_pool_miss_total Number of times a connection was not found in the pool.
# TYPE gitlab_redis_pool_miss_total counter
gitlab_redis_pool_miss_total{name="lhollinda-instance"} 1
# HELP gitlab_redis_pool_timeout_total Number of times a timeout occurred when looking for a connection in the pool.
# TYPE gitlab_redis_pool_timeout_total counter
gitlab_redis_pool_timeout_total{name="lhollinda-instance"} 0
# HELP promhttp_metric_handler_errors_total Total number of internal errors encountered by the promhttp metric handler.
# TYPE promhttp_metric_handler_errors_total counter
promhttp_metric_handler_errors_total{cause="encoding"} 0
promhttp_metric_handler_errors_total{cause="gathering"} 0
Edited by Luke Hollinda

Merge request reports