Reduce ConcurrencyLimit::WorkerExecutionTracker lease log level to DEBUG
<!--IssueSummary start--> <details> <summary> Everyone can contribute. [Help move this issue forward](https://handbook.gitlab.com/handbook/marketing/developer-relations/contributor-success/community-contributors-workflows/#contributor-links) while earning points, leveling up and collecting rewards. </summary> - [Label this issue](https://contributors.gitlab.com/manage-issue?action=label&projectId=278964&issueIid=588702) </details> <!--IssueSummary end--> <!--- Please read this! Before opening a new issue, make sure to search for keywords in the issues filtered by the "regression" or "type::bug" label: - https://gitlab.com/gitlab-org/gitlab/issues?label_name%5B%5D=regression - https://gitlab.com/gitlab-org/gitlab/issues?label_name%5B%5D=type::bug and verify the issue you're about to submit isn't a duplicate. ---> ### Summary After upgrading to GitLab 18.5+, Sidekiq nodes generate excessive INFO-level log entries in application_json.log with the message "Cannot obtain an exclusive lease. There must be another instance already in execution." ### Steps to reproduce ``` 1. Deploy GitLab 18.5+ with multiple Sidekiq nodes (e.g.,reference architecture with 2+ Sidekiq nodes) 2. Run some jobs 3. Monitor application_json.log on any Sidekiq node ``` ### Example Project <!--If possible, please create an example project here on GitLab.com that exhibits the problematic behavior, and link to it here in the bug report. If you are using an older version of GitLab, this will also determine whether the bug is fixed in a more recent version.--> ### What is the current _bug_ behavior? The cleanup_stale_trackers method runs after every Sidekiq job completion and attempts to acquire an exclusive lease. Apart from one thread across the cluster that holds this lease per worker type, all other threads log an INFO-level message. ### What is the expected _correct_ behavior? The log level should be DEBUG rather than INFO to avoid flooding logs with non-actionable messages. ### Relevant logs and/or screenshots ``` { "severity": "INFO", "time": "2026-02-02T09:25:10.068Z", "meta.caller_id": "Geo::VerificationStateBackfillWorker", "correlation_id": "...", "meta.root_caller_id": "Cronjob", "meta.feature_category": "geo_replication", "message": "Cannot obtain an exclusive lease. There must be another instance already in execution.", "lease_key": "sidekiq:concurrency_limit:{geo/verification_state_backfill_worker}", "class_name": "Gitlab::SidekiqMiddleware::ConcurrencyLimit::WorkerExecutionTracker", "lease_timeout": 600 } ``` ### Output of checks <!--If you are reporting a bug on GitLab.com, uncomment below--> <!--This bug happens on GitLab.com--> <!--and uncomment below if you have /label privileges--> <!--/label ~"reproduced on GitLab.com"--> <!--or follow up with an issue comment of `@gitlab-bot label ~"reproduced on GitLab.com"` if you do not--> #### Results of GitLab environment info <!--Input any relevant GitLab environment information if needed.--> <details> <summary>Expand for output related to GitLab environment info</summary> <pre> (For installations with omnibus-gitlab package run and paste the output of: \`sudo gitlab-rake gitlab:env:info\`) (For installations from source run and paste the output of: \`sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production\`) </pre> </details> #### Results of GitLab application Check <!--Input any relevant GitLab application check information if needed.--> <details> <summary>Expand for output related to the GitLab application check</summary> <pre> ```null gitlab-ee 18.6.3 Component Installed Version Version GUID -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- alertmanager v0.28.1 git:b2099eaa2c9ebc25edb26517cb9c732738e93910 bzip2 1.0.8 sha512:083f5e675d73f3233c7930ebe20425a533feedeaaa9d8cc86831312a6581cefbe6ed0d08d2fa89be81082f2a5abdabca8b3c080bf97218a1bd59dc118a30b9f3 cacerts 2022.07.19 sha256:6ed95025fba2aef0ce7b647607225745624497f876d74ef6ec22b26e73e9de77 chef-acme v4.1.8 git:8011acb376c72c42c77f50df6b729c068e4573e3 chef-gem 18.3.0 compat_resource v12.19.1 git:e36200f6b804915b68a4ce74c8b7a293c041d9fe config_guess c9092d05347c925a26f6887980e185206e13f9d6 git:39d10c4694fa0f1f2314c3542126698176b0d0fd consul v1.21.4 git:59b8b905d5abc3f955b81d76c138bf5eab2ca01a cosign v2.5.3 git:488ef8ceed5ab5d77379e9077a124a0d0df41d06 curl curl-8_15_0 git:97ae9ec8efe05bd0d82bbf9ee127c46bfae40c8c exiftool 12.99 git:4981552ec9bf94a0b5a64a06919b5e4f797c208e fast-stats v0.8.5 git:05463ec2553f1719ae023766ffb8764c86e47fe1 git v18.6.3 git:b5ccb6dc41c57b28d7962ecc39a59482e6e679e0 git-filter-repo v2.47.0 gitaly v18.6.3 git:b5ccb6dc41c57b28d7962ecc39a59482e6e679e0 gitlab-backup-cli 11bd2a07e00a1c1de5d6a7dfe4301fe4 gitlab-config-template 18.6.3 gitlab-cookbooks 18.6.3 gitlab-ctl 18.6.3 gitlab-ctl-ee 18.6.3 gitlab-elasticsearch-indexer v5.10.1 git:b30437c9530f928f861d20838da9bbc951bb9258 gitlab-exporter 15.6.0 gitlab-geo-psql 829ff725af11406efef31cfe6255afcb gitlab-healthcheck 0060154a010637bec197fb38217ddcbc gitlab-kas v18.6.3 git:f78d3aa8770f86fa1d1f4d9c2b2c56a786a818bc gitlab-pages v18.6.3 git:4c7173145d5b30a555ee3d48e6c9d03074f6549e gitlab-pg-ctl 708eef37ad59a61728d790c485aae4a6 gitlab-psql 4ac3cc368e73f3750591493104798fce gitlab-rails v18.6.3-ee git:7cea882041755dd12e4e524d83af83bb609a2fa0 gitlab-redis-cli e16c5ba796580cf0c774589f65392f65 gitlab-scripts 18.6.3 gitlab-selinux 18.6.3 gitlab-shell v14.45.3 git:f5da5793dc9b9d720626f3f59a61a6331d236305 gitlabsos ccb1edb6aafe88e1ba612643c015f41471bbab88 git:25547c1702320af7992a380700958f384c89f67f gnupg 2.2.41 sha256:13f3291007a5e8546fcb7bc0c6610ce44aaa9b3995059d4f8145ba09fd5be3e1 go-crond 23.12.0 git:4f2e74e2a5544a72c6523c22f687d72be593e981 gpgme 1.18.0 sha256:361d4eae47ce925dba0ea569af40e7b52c645c4ae2e65e5621bf1b6cdd8b0e9e graphicsmagick 1.3.46 sha256:c7c706a505e9c6c3764156bb94a0c9644d79131785df15a89c9f8721d1abd061 inspec-gem 6.6.0 jemalloc 5.3.0 git:1972241cd204c60fb5b66f23c48a117879636161 krb5 krb5-1.22.1-final git:081d329beae207ee8110bb18bb2e938d3c683568 libarchive 3.8.2 sha256:5f2d3c2fde8dc44583a61165549dc50ba8a37c5947c90fc02c8e5ce7f1cfb80d libassuan 2.5.5 sha256:8e8c2fcc982f9ca67dcbb1d95e2dc746b1739a4668bc20b3a3c5be632edb34e4 libedit 20150325-3.1 sha256:c88a5e4af83c5f40dda8455886ac98923a9c33125699742603a88a0253fcc8c5 libevent release-2.1.12-stable git:5df3037d10556bfcb675bc73e516978b75fc7bc7 libffi 3.2.1 sha256:d06ebb8e1d9a22d19e38d63fdb83954253f39bedc5d46232a05645685722ca37 libgcrypt 1.9.4 sha256:ea849c83a72454e3ed4267697e8ca03390aee972ab421e7df69dfe42b65caaf7 libgpg-error 1.56 sha256:82c3d2deb4ad96ad3925d6f9f124fe7205716055ab50e291116ef27975d169c0 libiconv 1.15 sha256:ccf536620a45458d26ba83887a983b96827001e92a13847b45e4925cc8913178 libicu release-63-1 git:227c729b0e9e1d4e15993741a8854180a42f998a libjpeg-turbo 2.1.5.1 git:8ecba3647edb6dd940463fedf38ca33a8e2a73d1 libksba 1.6.3 sha256:3f72c68db30971ebbf14367527719423f0a4d5f8103fc9f4a1c01a9fa440de5c liblzma 5.2.4 sha256:b512f3b726d3b37b6dc4c8570e137b9311e7552e8ccbab4d39d47ce5f4177145 libossp-uuid 1.6.2 sha256:11a615225baa5f8bb686824423f50e4427acd3f70d394765bdff32801f0fd5b0 libpng v1.6.51 git:49363adcfaf098748d7a4c8c624ad8c45a8c3a86 libtensorflow_lite v2.6.0 git:96cd1956e607ca2b256ac01444caf3a9c61396fa libtiff v4.7.1 git:5fe20d0e9aba49a6a350ed533459d1505203838f libtool 2.4.6 sha256:e3bd4d5d3d025a36c21dd6af7ea818a2afcd4dfc1ea5a17b39d7854bcd0c06e3 libxml2 2.14.5 sha256:03d006f3537616833c16c53addcdc32a0eb20e55443cba4038307e3fa7d8d44b libxslt 1.1.43 sha256:5a3d6b383ca5afc235b171118e90f5ff6aa27e9fea3303065231a6d403f0183a libyaml 0.2.5 sha256:c642ae9b75fee120b2d96c712538bd2cf283228d2337df2cf2988e3c02678ef4 logrotate 3.22.0 git:41efb71b765b08e53e2c411e0a2897d30f44eefc mattermost 10.11.8 sha256:90138a2b4b6230822eb285731befa40c58c67aa987c9cec11d162c59ac56f492 ncurses 6315e1a380ecdb706d4f6518d2e8c7eb0db8fbe2 git:87c2c84cbd2332d6d94b12a1dcaf12ad1a51a938 nginx release-1.29.0 git:235f409907fd60eb2d8f6ecdc0e5cb163dd6d45f nginx-module-vts v0.2.4 git:bdb2699d87a84ed593de3ca114290740b530a514 ngx_security_headers 0.0.9 git:5f34cd8395614caf7650a592c89589a2ced4db02 node-exporter v1.9.1 git:f2ec547b49af53815038a50265aa2adcd1275959 npth 1.6 sha256:1393abd9adcf0762d34798dc34fdcf4d0d22a8410721e76f1e3afcd1daa4e2d1 omnibus-ctl v0.6.12 git:4e968f4cf713a545bbb70ee13b8ff7ebd37f7f3e omnibus-gitlab-gems 20250418 path:/builds/gitlab/omnibus-gitlab/config/templates/omnibus-gitlab-gems openssl openssl-3.4.1 git:2466b57b893503f602622ad4f7cba1f3a2585ec0 package-scripts 18.6.3+ee.0 patroni 3.0.1 pcre2 pcre2-10.45 git:d213fba6c8ecf7b469ea32368205be09c8e0073f pgbouncer pgbouncer_1_24_1 git:2305be8887b902c46bec7acdbf644e039061195c pgbouncer-exporter v0.11.0 git:766ca7ef118144cfe872c096af7c638ac3c769d6 pkg-config-lite 0.28-1 sha256:21b76ec4e115ee30f9b3077a2506e48e8b837332ed4d30c9776502e69c6a29e5 popt 1.16 sha256:e728ed296fe9f069a0e005003c3d6b2dde3d9cad453422a10d6558616d304cc8 postgres-exporter v0.15.0 git:68c176b8833b7580bf847cecf60f8e0ad5923f9a postgresql 16.10 sha256:de8485f4ce9c32e3ddfeef0b7c261eed1cecb54c9bcd170e437ff454cb292b42 prometheus v3.6.0 git:491734606591ef8c188f5489c70380805bc868c0 psycopg2 2.9.10 git:a805acf59f402f554e95624b5e27518169ca7715 python-docutils 0.19 python3 3.9.24 sha256:9a32cfc683aecaadbd9ed891ac2af9451ff37f48a00a2d8e1f4ecd9c2a1ffdcb rb-readline master git:9fba246073f78831b7c7129c76cc07d8476a8892 redis 7.2.11 git:0f39801756b21464b2fade19dbe213c67b293790 redis-exporter v1.79.0 git:6e630b691ce2627dd3fcf1194d4e9d1dd24e689f registry v4.31.1-gitlab git:3a6feee7b98d71f0b67230bcb8c1965c4c03a181 rsync 3.4.1 sha256:2924bcb3a1ed8b551fc101f740b9f0fe0a202b115027647cf69850d65fd88c52 ruby 3.2.8 sha256:77acdd8cfbbe1f8e573b5e6536e03c5103df989dc05fa68c70f011833c356075 rubygems 3.7.1 runit 2.1.2 sha256:6fd0160cb0cf1207de4e66754b6d39750cff14bb0aa66ab49490992c0c47ba18 spam-classifier 0.3.0 sha256:c9f7e147d195a45e32c35765e138e006e7636218f8c4413e67d0cef5513335a8 spamcheck v0.3.2 git:e1d8002ff4fbff2f5b0c8baa6d381607a2863f7f unzip 6.0.29 sha256:036d96991646d0449ed0aa952e4fbe21b476ce994abc276e49d30e686708bd37 version-manifest 0.0.1 zlib-ng 2.2.4 git:e2ecd2665b8ddf5eadae6925c0f0c2f2da86fed3 ``` </pre> </details> ### Related MRs: \- https://gitlab.com/gitlab-org/gitlab/-/merge_requests/205730+ \- https://gitlab.com/gitlab-org/gitlab/-/merge_requests/206685+ <!--If you don't have /label privileges, follow up with an issue comment of `@gitlab-bot label ~"type::bug"`-->
issue