perf(collector): stream data-file writes to cap serialization memory

Andrew Dunnrequested to merge
perf/stream-collector-output into main

Problem

The per-resource collectors wrote their output files through writeJSON, which called json.MarshalIndent on the whole slice. That built the entire indented JavaScript Object Notation (JSON) blob in one []byte on top of the already-in-memory slice: a peak-memory doubling at serialization time. On a large instance (1000+ users, 10k+ events) that blob is the memory ceiling.

Fix

Add writeEnvelopeStream: it writes the metadata header, then streams the data array element-by-element through a json.Encoder over a bufio.Writer. Peak serialization memory is now one encoded record plus the buffer instead of the whole file. The write stays atomic (temp file + rename), so a crash mid-write never corrupts the target.

Routed the slice-payload envelopes through it: events, merge_requests, issues, pipelines, roster, identities, including the incremental merge-rewrite path. manifest.json and watermark.json keep the buffered path (small, non-slice payloads).

Scope

  • In scope: the serialization buffer.
  • Out of scope: the in-memory collection set that dedup/merge requires. That the full deduped slice lives in memory before write is inherent to dedup, not changed here. Collection, dedup, and merge semantics are untouched.

Escaping and indentation

Escaping matches the old json.MarshalIndent path (HyperText Markup Language (HTML) escaping on). Indentation uses the same two-space style. Byte-identical pretty-print is not required: the analyzer reads via json.NewDecoder, the incremental loaders via json.Unmarshal. Both parse the output.

Tests

  • Round-trip through the incremental loader and the analyzer's json.NewDecoder path: decoded slice equals input (order + content), proving analyzer compatibility.
  • Well-formed: empty slice -> [], single element, multiple elements with correct commas and no trailing comma; json.Valid on every case.
  • Escaping cross-checked against json.Marshal (derived at runtime, no hardcoded escape form).
  • Determinism: same input -> byte-identical output.
  • Nested pipelines round-trip (jobs slice).

Analyzer and renderer suites pass against the streamed output. go vet clean, go test -race passes, golangci-lint 0 issues, coverage 69.8%.

Merge request reports