## Summary The bufferer goroutine of `eventQueue` in `notifications/sinks.go` never blocks while it holds events that the sender goroutine has not taken yet. The `select` of its main loop has an empty `default:` case, so when the sender is still busy writing the previous event and no new event arrives, the `select` returns at once and the loop starts over. That happens whenever an event waits behind a write in flight, and while a notification endpoint is slow or unreachable it lasts as long as the endpoint stays that way, because the backoff sink keeps the sender busy through every retry. Each endpoint has its own queue, so each endpoint in that state keeps one CPU core busy. The loop that drains the queue when the registry shuts down has the same empty `default:`, so it spins too, for up to `queuepurgetimeout`. ## Steps to reproduce 1. From a checkout of `master`, build the registry and start it with `inmemory` storage, no database, and one notification endpoint on a port nothing listens on: ```shell go build -o /tmp/registry ./cmd/registry cat > /tmp/config.yml <<'EOF' version: 0.1 log: level: error storage: inmemory: {} database: enabled: false http: addr: 127.0.0.1:5000 notifications: endpoints: - name: down url: http://127.0.0.1:9/ timeout: 1s backoff: 1s maxretries: 10 EOF /tmp/registry serve /tmp/config.yml > /tmp/registry.log 2>&1 & pid=$! sleep 2 ``` 1. Push a blob and pull it three times, so that events for the endpoint wait in its queue while the sender retries the first one: ```shell printf hello > /tmp/blob digest=sha256:$(sha256sum /tmp/blob | cut -d' ' -f1) loc=$(curl -s -D - -o /dev/null -X POST http://127.0.0.1:5000/v2/foo/bar/blobs/uploads/ | tr -d '\r' | awk 'tolower($1)=="location:" {print $2}') curl -s -o /dev/null -w '%{http_code}\n' -X PUT "$loc&digest=$digest" -H 'Content-Type: application/octet-stream' --data-binary @/tmp/blob for i in 1 2 3; do curl -s -o /dev/null -w '%{http_code}\n' "http://127.0.0.1:5000/v2/foo/bar/blobs/$digest"; done ``` 1. Read the CPU time the registry process uses over the next five seconds: ```shell sleep 1 before=$(awk '{print $14+$15}' /proc/$pid/stat); sleep 5; after=$(awk '{print $14+$15}' /proc/$pid/stat) echo "CPU time in 5 s of wall time: $(( (after - before) * 1000 / $(getconf CLK_TCK) )) ms" ``` ## What is the current bug behavior? On `master` at 2f9d3333 (Linux amd64, Go 1.26.8) the steps print: ```plaintext 201 200 200 200 CPU time in 5 s of wall time: 5020 ms ``` The registry keeps a full core busy while it only waits for the endpoint. With a second endpoint like the first, on another port nothing listens on, three samples of 5 seconds read 9990, 9630 and 9950 ms: one core per endpoint. ## What is the expected correct behavior? While the sender is busy and no new event arrives, the queue waits without using CPU until a new event comes in, the sender is ready for the next one, or the queue is closed. ## Relevant code At 2f9d3333: - [`notifications/sinks.go#L299-324`](https://gitlab.com/gitlab-org/container-registry/-/blob/2f9d3333918d394bff04cea9e4f1637792d4ebd0/notifications/sinks.go#L299-324): the `select` of the main loop of `(*eventQueue).bufferer`, with the empty `default:` at line 323. It runs while the queue holds at least one event; with an empty queue the bufferer blocks in the `select` at lines 287-295. - [`notifications/sinks.go#L334-344`](https://gitlab.com/gitlab-org/container-registry/-/blob/2f9d3333918d394bff04cea9e4f1637792d4ebd0/notifications/sinks.go#L334-344): the loop that drains the queue after `Close`, with the same empty `default:` at line 342. - [`notifications/endpoint.go#L172-186`](https://gitlab.com/gitlab-org/container-registry/-/blob/2f9d3333918d394bff04cea9e4f1637792d4ebd0/notifications/endpoint.go#L172-186): each endpoint puts its queue in front of a backoff sink, which retries a failed write with exponential backoff before the sender can take the next event. Both empty `default:` cases came with 4d0e4639 (fix: make eventqueue sink interruptable, 2024-09-25), first released in v4.10.0-gitlab, and v4.42.0-gitlab still has both. ## Possible fixes Remove both empty `default:` cases, so that each `select` blocks until one of its channels is ready. Neither loop has anything else to do while it waits: the main loop still selects between a new event, the sender and `doneCh`, and the drain loop is still bounded by the purge timer. With that change the steps above print `CPU time in 5 s of wall time: 0 ms`. The `respects exact queue size limit` subtest of `TestEventQueueMaxSize` does not wait for the sender to take the first event before it fills the queue. It passes on `master`, and fails in some runs once the bufferer blocks (between 3 and 15 of 50 runs in my measurements, depending on the machine), so it needs to wait for the sender explicitly. I have a merge request ready for both.