Filter API keys from LLM logs

Shiyuan Chenrequested to merge
gitlab-jh/jh-team/gitlab:fix-629564-redact-llm-api-keys into master

What does this MR do and why?

Expanded AI logging can write self-hosted model credentials from model_metadata.api_key into llm.log. Filter structured api_key fields recursively when the LLM logger serializes an entry, replacing their values with [FILTERED] while preserving diagnostic context.

Filtering does not mutate the original payload, so outgoing requests retain their authentication credentials. The regression specs cover all four log levels, the instance/feature flag/namespace expanded logging switches, AI Gateway Client normal and streaming requests, and Duo Chat StepExecutor requests.

Two existing legacy logger fields exercised by the new specs are registered in the Labkit logging baseline. This change filters structured fields; it does not parse credentials embedded in arbitrary text.

References

Closes https://gitlab.com/gitlab-org/gitlab/-/issues/629564

The issue is labeled security-fix-in-public, and the maintainer confirmed that this fix can be contributed to the canonical repository: https://gitlab.com/gitlab-org/gitlab/-/work_items/629564#note_3882735325

How to set up and validate locally

Run:

bundle exec rspec ee/spec/lib/gitlab/llm/logger_spec.rb ee/spec/lib/gitlab/llm/ai_gateway/client_spec.rb ee/spec/lib/gitlab/duo/chat/step_executor_spec.rb

Local validation of this commit:

  • 132 examples, 0 failures; exit status 0.
  • Regression check with the new serialization method removed only in the test process: 15 examples, 13 expected failures exposing the synthetic API key.
  • RuboCop: four Ruby files, no offenses.
  • Gitleaks: final commit scanned, no leaks.

The request-path specs verify both redacted log output and unchanged outgoing request credentials.

MR acceptance checklist

Reviewed against the MR acceptance checklist. Independent local code review completed without blocking findings. CI and upstream review remain pending.

Merge request reports