GITLABCOM-ESTV mixes deploy-signal kills with real indexer failures

Summary

Sentry issue GITLABCOM-ESTV (Ai::ActiveContext::Code::IndexerBase::Error) is at high and growing volume: 2.6k events over 30 days, around 26 a day, still active today.

gitlab-org/gitlab#627763 triaged this issue in September and marked it as accepted noise, tracked under the design work in #548766 (closed). That conclusion deserves a second look. Sentry groups exceptions by class, and this class bundles at least two unrelated causes together. Neither one is "just noise" on closer inspection.

What's actually in this bucket

1. The indexer subprocess gets killed by a deploy, and we misreport it as a real failure.

Ai::ActiveContext::Code::Indexer shells out to the gitlab-elasticsearch-indexer binary. A gitlab.com deploy sends SIGTERM to the whole Sidekiq process group (Gitlab::SidekiqSignals), and that signal also kills this subprocess, since it's a child process in the same group. Today, Indexer#run treats any non-zero exit the same way, whether the binary hit a real bug or a deploy just interrupted it. Two consequences:

  • The exception message can be actively misleading. The error_details field is just the last line the subprocess wrote to stderr before it died, and that's often a benign INFO log (for example "Elasticsearch connection successful"), not the real cause.
  • InitialIndexingService and IncrementalIndexingService treat a deploy interruption as a real failure, and can flip a repository row to pending_retry or failed for no actual reason. A repo whose indexing routinely outlives the gap between deploys could keep failing for reasons that have nothing to do with its own data.

2. Elasticsearch connectivity and shard failures, unrelated to deploys.

A separate slice of this same bucket is Elastic::TimeoutError, Faraday::ConnectionFailed, and "all shards failed" style errors, on the initial ES/OpenSearch/Postgres connection check inside the indexer binary. The binary itself has no retry or backoff logic anywhere for these connection attempts or for bulk-indexing calls. #548766 (closed) already tracks a related gap (partial bulk failures that silently return success), but the connection-level "all shards failed" case isn't covered by that.

To investigate

  • Confirm current volume split between cause 1 and cause 2 in the last 30 days of GITLABCOM-ESTV events.
  • For cause 1: can Indexer#run distinguish "subprocess died from a deploy signal" from "subprocess hit a real bug", so a deploy doesn't corrupt repository state or produce a misleading Sentry event? A first look suggests this is possible with a small, contained change in lib/gitlab/popen.rb and ee/app/services/ai/active_context/code/, without touching Sidekiq or Redis internals.
  • For cause 2: file a follow-up against gitlab-elasticsearch-indexer for the missing retry/backoff on the initial connection check, since that's a fix that belongs in that repo, not here.
  • Decide whether a repository that gets interrupted by a deploy on every single retry attempt, forever, is worth guarding against, or whether that's rare enough to ignore.