archive: let hide-secrets cache what it learns from source copies

Part 3 of 3 for TFT-5159. Merge in order:

  1. !1102 (merged) hide-secrets: add a constant-memory stream implementation
  2. !1103 (merged) hide-secrets: skip files already verified clean
  3. !1104 (merged) archive: let hide-secrets cache what it learns from source copies (this MR)

Depends on !1103 (merged) and targets its branch, so the diff shows this MR's commits only. GitLab retargets it to main once !1102 (merged) and !1103 (merged) merge.

For the progress and execute stages, and for the sync before the final report, archive copies each source, redacts the copy, uploads it, and deletes it. hide-secrets used to cache what it learned under the copy's temporary path, which never comes back, so every tick scanned every file from scratch.

This MR passes cache_path=original_source and source_stats to the hide_secrets call in archive_stage() whenever it redacts a copy. prepare_source_copy() opens each regular file of the original once, copies the data from that descriptor, and stats the descriptor before and after. It keeps the stat only when the two match. A file which changes during the copy gets no stat, and hide-secrets scans its next copy again. The same goes for a file changed less than CACHE_SETTLE_TIME seconds before the copy, and for a symlink or special file, which shutil.copy2() copies as before.

Both stats come from the descriptor rather than the path. Someone who swaps a parent directory away and back around the open could otherwise get another file's clean bytes recorded against the original.

The next tick copies the same original again, and hide-secrets skips the files which have not changed since. The copy itself stays, as does everything else about how archive syncs.

Generated-by: Claude Code

Edited by Miroslav Vadkerti

Merge request reports

Loading
Loading