fix(collector): use real incremental window and cover CI-active projects

Andrew Dunnrequested to merge
fix/collector-incremental-coverage into main

What

Two data-correctness fixes in internal/collector.

1. Real incremental window (headline)

orchestrator.go read the previous watermark.LastCollectedAt but never used it. Every per-resource collector (events, merge requests, issues, pipelines) calls cfg.window(), which unconditionally went back WindowDays from now. So an incremental run refetched the entire window and only de-duped afterward, paying full API cost and defeating the feature.

The orchestrator now threads the watermark into the collection window start via cfg.withCollectSince(watermark). The start becomes watermark - IncrementalOverlap, so collectors fetch only records since the last run.

  • Safety overlap: re-collect a one-day buffer before the watermark to catch late-arriving or edited records. The existing dedup/merge layer (merge.go) absorbs the overlap, so no record is double-counted.
  • A full run with no watermark still uses WindowDays.
  • The incremental start never predates the WindowDays floor, so a long gap between runs falls back to the full window rather than an unbounded backfill.

2. CI-only project coverage

deriveProjectIDs built the pipeline-collection set only from projects with an in-window merge request or issue. A project with active CI but no in-window MR or issue contributed zero pipelines, silently undercounting CI/CD and Security.

The orchestrator now also seeds the set from a lightweight projects sweep, GET /api/v4/projects?last_activity_after=<window start>&simple=true (paginated via the existing Link-header pagination), and unions it with the MR/issue-derived set. The sweep is best-effort: a failure logs and falls back to the MR/issue set rather than aborting the run.

Tests

In internal/collector/coverage_window_test.go:

  • Incremental window start reflects the watermark minus overlap (captures the created_after query param the collector sends). Without a watermark it is now - WindowDays.
  • The projects sweep covers a CI-active project with no MR or issue, asserted at the union helper and end-to-end through Run (pipelines fetched for the swept-only project).

Validation

  • go vet ./... clean.
  • CGO_ENABLED=1 go test -race ./... passes.
  • golangci-lint run reports 0 issues.
  • Total coverage 70.0 percent; collector package 85.9 percent.
  • Determinism preserved: project-set ordering follows the same map-iteration pattern deriveProjectIDs already used; artifact content is keyed and merged order-independently.

Merge request reports