## Summary After the labkit 2.0.0 bump (gitlab-org/gitlab!236506) introduced an atomic Lua `INCRBY + TTL` script in `Labkit::RateLimit::Evaluator`, CI test jobs that run against Redis Cluster (host `redis://rediscluster:7003`) started emitting `RedisClient::NoScriptError` exceptions. Pre-bump pipelines produced 0; post-bump pipelines produced 3. Reported by @jay_mccure in gitlab-org/gitlab!236506 (note 3383644277): https://gitlab.com/gitlab-org/gitlab/-/merge_requests/236506#note_3383644277 | Pipeline | Pipeline run time | NoScriptError count | |---|---|---| | Pre-labkit-2.0 (07:17 UTC May 18, [pipe 2533138884](https://gitlab.com/gitlab-org/gitlab/-/pipelines/2533138884), labkit 1.19.0) | 36 min | 0 | | Post-labkit-2.0 (10:11 UTC May 18, [pipe 2533756573](https://gitlab.com/gitlab-org/gitlab/-/pipelines/2533756573), labkit 2.0.0) | 33 min | 3 | ## Observed exception ``` exception.class: RedisClient::NoScriptError exception.message: NOSCRIPT No matching script. Please use EVAL. (redis://rediscluster:7003) extra.storage: rate_limiting extra.storage_shard: default caller_id: GraphqlController#execute ``` Stack trace (abridged) — bubbles up from `evalsha` through Labkit and surfaces at the GraphQL resolver: ``` redis (5.4.1) lib/redis/commands/scripting.rb:110:in `_eval' redis (5.4.1) lib/redis/commands/scripting.rb:97:in `evalsha' gitlab-labkit (2.0.0) lib/labkit/redis/script.rb:35:in `eval' gitlab-labkit (2.0.0) lib/labkit/rate_limit/evaluator.rb:195:in `eval_incr_script' gitlab-labkit (2.0.0) lib/labkit/rate_limit/evaluator.rb:171:in `block in incr_with_ttl' ... lib/gitlab/application_rate_limiter/labkit_adapter.rb:72:in `run!' lib/gitlab/application_rate_limiter.rb:369:in `dispatch_to_labkit' ``` Full trace is in the linked MR note. ## Diagnostic hypotheses to verify This is an investigation issue — the diagnosis below is a starting point, not a committed fix. 1. **Rescue catches the wrong exception class.** `Labkit::Redis::Script#eval` in labkit-ruby rescues `::Redis::CommandError` and checks `e.message.start_with?("NOSCRIPT")`: https://gitlab.com/gitlab-org/ruby/gems/labkit-ruby/-/blob/master/lib/labkit/redis/script.rb#L34-40 The reported exception is `RedisClient::NoScriptError`, a distinct class from `Redis::CommandError`. The high-level `redis` gem normally re-wraps lower-level errors, but the `redis-clustering` scripting path appears to surface `RedisClient::NoScriptError` directly. If the rescue does not catch it, the `EVAL` fallback never runs. 2. **Redis Cluster cross-node script cache.** Even with the rescue fixed, `SCRIPT LOAD` only populates the script cache on a single primary. When `EVALSHA` is later routed by key slot to a different primary that has not seen the script, `NOSCRIPT` recurs. The current single-connection `EVAL` fallback masks the next request but does not prime the rest of the cluster — verify whether the rate-limiting Redis instance in CI is a true cluster or a single-shard `rediscluster:7003` pointed at one node. 3. **CI-only vs production behaviour.** The `extra.storage: rate_limiting, extra.storage_shard: default` tags indicate this is happening on the rate-limiting shard. Confirm whether the same trace can happen on the production rate-limiting shard (likely yes) or only in the CI Redis Cluster topology. ## Proposed investigation steps - [ ] Reproduce locally with a Redis Cluster topology (e.g. via the GDK rate-limiting shard configured as a cluster) by calling `Labkit::RateLimit::Limiter#check` and then forcing `SCRIPT FLUSH` between calls. - [ ] Confirm which exception class is raised — is the `Redis::CommandError` rescue dead code, or does it catch the simple-Redis case while the cluster path leaks through? - [ ] Decide the right fix: widen the rescue to also catch `RedisClient::NoScriptError` and/or rework the script-loading strategy to be cluster-aware. - [ ] Check whether `INCR_SCRIPT` and `SADD_SCRIPT` are equally affected (both call through the same `Labkit::Redis::Script#eval`). - [x] Quantify production exposure on GitLab.com — search Sentry for `RedisClient::NoScriptError` since 2026-05-18. ## Related - MR that introduced the regression: gitlab-org/gitlab!236506 - Upstream labkit MR: gitlab-org/ruby/gems/labkit-ruby!291 - Files of interest in labkit-ruby: - `lib/labkit/redis/script.rb` (the rescue clause) - `lib/labkit/rate_limit/evaluator.rb` (`INCR_SCRIPT` and `SADD_SCRIPT` call sites at lines ~219 and ~248) /cc @jay_mccure (reporter), @reprazent (MR author) 🤖beep boop