hide-secrets: add a constant-memory stream implementation
Part 1 of 3 for TFT-5159. Merge in order:
- !1102 (merged) hide-secrets: add a constant-memory
streamimplementation (this MR) - !1103 (merged) hide-secrets: skip files already verified clean
- !1104 (merged) archive: let hide-secrets cache what it learns from source copies
sed -z splits records on NUL bytes. Log files contain none, so sed loads each file into one pattern space and peaks at twice its size. A 99-hour reservation grew its guest console log to several GiB, and redacting it during archiving pushed the worker past the oom limit. Past 2 GiB sed gives up with regex input buffer length larger than INT_MAX, after it has already allocated the memory.
This MR adds an implementation option. The default, sed, runs today's code, moved unchanged into _hide_secrets_sed(). The opt-in stream reads each file in 1 MiB chunks and holds back enough of each read to catch a secret split across reads:
- single-line secrets match as plain bytes, with no escaping at all
- a multi-line secret compiles from its pieces of text and the whitespace runs with line breaks between them. Each run becomes
(?:[^\S\n]*\n){N}\s*, which matches any amount of whitespace with at least as many line breaks, mirroring the[[:space:]]*padding of thesedpath, and matches a given run in one way only, so it never backtracks - whitespace around a multi-line secret is not part of it, so
streamredacts a superset of whatseddoes - overlapping matches, of one secret or several, are redacted together into one
hidden, wherever reads split the file - a file without secrets is only read, keeping its inode and mtime
- a file with secrets goes through a temporary file in the same directory,
fsync, and a rename, followed by anfsyncof the directory
On ordinary text stream holds back about as many bytes as the longest secret has. Only the first line of a secret followed by more than 16 MiB of whitespace can push it further, and such a file fails instead of risking a leak.
On a 1 GiB console log with an indented PEM key, stream finishes in 1.6 seconds at 85 MiB peak RSS, and a 256 MiB file of blank lines takes 0.4 seconds. On a 3.1 GiB console log sed grows to 6.4 GiB and fails.
The existing tests now run against both implementations, and a parity test checks both produce identical output for the multi-line shapes which differ between POSIX and Python regular expressions. Further tests cover overlapping secrets at every read offset, indentation up to 2 MiB, blank lines in a secret, and the 16 MiB limit.
Rollout
sed stays the default, so merging this changes nothing on the workers. The OOM goes away once the deployment config sets implementation: stream for the hide-secrets module. Setting it back to sed restores today's behaviour without a code change.
Generated-by: Claude Code