Cap refs logged on ActiveContext batch failures

Arturo Herrerorequested to merge
606544-safer-activecontext-error-logs into master

Description

Follow-up to the logging improvements in !246009 (merged).

When a batch fails in ActiveContext, the error log includes every serialized ref in the batch. A full batch holds 1,000 refs, and the log line can exceed log pipeline limits. The pipeline then drops the line. This hides exactly the failures that matter most. On GitLab.com we measured that lines with up to ~957 refs reach Kibana, and lines with ~989 or more do not. See the investigation in #606544 (comment 3710630691).

Changes:

  • Batch failure logs include refs_count and a sample of 10 refs (refs_sample) instead of the full refs array. The line stays small, so it is always indexed.
  • The bulk_exception log (the whole bulk request to Elasticsearch raises) includes queue_name and refs_count. Before, this line could represent 1 or 1,000 failed items, and we could not tell which queue it came from.

There is no behavior change. The same log lines fire at the same points.

Related to #606544 (closed).

Merge request reports