hide-secrets: add a constant-memory stream implementation

Part 1 of 3 for TFT-5159. Merge in order:

  1. !1102 (merged) hide-secrets: add a constant-memory stream implementation (this MR)
  2. !1103 (merged) hide-secrets: skip files already verified clean
  3. !1104 (merged) archive: let hide-secrets cache what it learns from source copies

sed -z splits records on NUL bytes. Log files contain none, so sed loads each file into one pattern space and peaks at twice its size. A 99-hour reservation grew its guest console log to several GiB, and redacting it during archiving pushed the worker past the oom limit. Past 2 GiB sed gives up with regex input buffer length larger than INT_MAX, after it has already allocated the memory.

This MR adds an implementation option. The default, sed, runs today's code, moved unchanged into _hide_secrets_sed(). The opt-in stream reads each file in 1 MiB chunks and holds back enough of each read to catch a secret split across reads:

  • single-line secrets match as plain bytes, with no escaping at all
  • a multi-line secret compiles from its pieces of text and the whitespace runs with line breaks between them. Each run becomes (?:[^\S\n]*\n){N}\s*, which matches any amount of whitespace with at least as many line breaks, mirroring the [[:space:]]* padding of the sed path, and matches a given run in one way only, so it never backtracks
  • whitespace around a multi-line secret is not part of it, so stream redacts a superset of what sed does
  • overlapping matches, of one secret or several, are redacted together into one hidden, wherever reads split the file
  • a file without secrets is only read, keeping its inode and mtime
  • a file with secrets goes through a temporary file in the same directory, fsync, and a rename, followed by an fsync of the directory

On ordinary text stream holds back about as many bytes as the longest secret has. Only the first line of a secret followed by more than 16 MiB of whitespace can push it further, and such a file fails instead of risking a leak.

On a 1 GiB console log with an indented PEM key, stream finishes in 1.6 seconds at 85 MiB peak RSS, and a 256 MiB file of blank lines takes 0.4 seconds. On a 3.1 GiB console log sed grows to 6.4 GiB and fails.

The existing tests now run against both implementations, and a parity test checks both produce identical output for the multi-line shapes which differ between POSIX and Python regular expressions. Further tests cover overlapping secrets at every read offset, indentation up to 2 MiB, blank lines in a secret, and the 16 MiB limit.

Rollout

sed stays the default, so merging this changes nothing on the workers. The OOM goes away once the deployment config sets implementation: stream for the hide-secrets module. Setting it back to sed restores today's behaviour without a code change.

Generated-by: Claude Code

Edited by Miroslav Vadkerti

Merge request reports

Loading
Loading