Don't log errors for Gitlab::ExclusiveLeaseHelpers::FailedToObtainLockError in Elasticsearch/Zoekt indexer workers
Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.
Problem
In our Elasticsearch indexer and Zoekt indexer we use a locking mechanism to ensure that we don't run 2 indexing processes in parallel. These locks are designed to raise an error if the project is already being indexed which then triggers Sidekiq to retry the job later.
This use of errors means that we fill our logs and Sidekiq UI with these errors. This would be fine if this was actionable but actually these errors are part of a perfectly healthy system and usually do not need any action. In fact we have this logged over 1000 times a day for GitLab.com.
This logging and errors is not really a major issue but it starts to cause confusion when customers are troubleshooting other issues related to Elasticsearch indexing. It can sometimes send them down a rabbit hole investigating something that is not necessarily a root cause of whatever is their main issue. The log/UI noise can also just be annoying or hide other issues.
Solution
Since this isn't really an error and it's not something that needs to be fixed we should not raise an exception. Instead we should just check if we can acquire the lock and if not then we should trigger a retry of the job without error (perhaps just requeue with some delay). This would preserve the existing behaviour without filling the logs with errors.