Seemingly innocuous error log message related to obtaining a lease
Summary
On Gitlab.com we see the following error log quite often from NOT our sidekiq fleet of servers:
Cannot obtain an exclusive lease. There must be another instance already in execution.
During a quick pairing session, it was discovered that we may be reusing a code path for sidekiq that helps us modify user attributes. For API calls, this ends up throwing an error message that can safely be ignored. Throwing an error message for a request that is ultimately successful is a bit strange and only invokes confusion among persons that are doing deep dive into logs.
This appears to be a problem found on GitLab.com's WEB and API fleet of servers.
Steps to reproduce
Make an HTTP request type: POST at /api/:version/projects/user/:user_id
Observe that our API will return an HTTP 201, indicating a successful request, but note an error in our application log. If we are indeed waiting for a lease, it seems kinda strange that we'll wait up to 10 minutes according to the classes method, however, we returned an HTTP 201 in .705 seconds on the example request I'm looking at. This suggests that we may be waiting for a lease unnecessarily for this code path?
Example Project
This can be seen via automatic QA runs in our staging environment.
This can be seen via log spelunking in our production environment.
What is the current bug behavior?
Either a log message can safely be ignored, which in this particular use case, doesn't seem like we should be hitting this code, OR we should maybe modify the log message such that it isn't an error?
What is the expected correct behavior?
Determined by the above investigation
Relevant logs and/or screenshots
In our staging environment: https://nonprod-log.gitlab.net/goto/43d02831b0b1bb27718914a7538c5797
In our production environment: https://log.gprd.gitlab.net/goto/d8a0c6ab34479db8260ccf0bd6365000