Page through trace chunks with a cursor in the archive cron

Max Fanrequested to merge
634314-archive-traces-cron-cursor into master

What does this MR do and why?

Ci::ArchiveTracesCronWorker times out on every run on GitLab.com, so trace archivals that failed in Ci::ArchiveTraceWorker are never retried.

Its query (Ci::Build.with_stale_live_trace + find_each) has to join every finished build that still has trace chunks (about 1.56M) before it can sort by p_ci_builds.id and pick 100. On a clone the first batch took 1,507 s, against a 15 s statement timeout.

This MR changes how Ci::ArchiveTraceService#batch_execute pages through candidates, behind the ci_archive_traces_cron_cursor flag:

  • Each batch reads a window of up to 100 distinct (build_id, partition_id) pairs from ci_build_trace_chunks after a cursor, then loads only those builds by primary key. Each query reads about 100 builds' worth of chunk rows, however many builds don't qualify.
  • The cursor is kept in Redis between runs and saved after every window, so each run continues where the previous one stopped, even if a run is interrupted. It expires after 7 days without a run, the same TTL as trace chunks in Redis: after that, builds it skipped have lost their logs anyway. When a window comes back empty, the cursor is cleared and the next run starts again from 0. That pass picks up builds whose backoff has expired.
  • Each pass ends at the newest build with chunks when it started (pass_end, saved with the cursor). New builds get chunks faster than a window is read, so without this bound a pass would follow them and never finish. Builds created during a pass are handled by the next one.
  • Builds on archived pipelines are skipped without counting toward the loop limit. Their trace metadata partitions may be detached, so they can't be archived (see 4edf9d80).
  • Each run logs its stats (processed, skipped_archived_pipeline, limit_reached, over_time, cursor_start, pass_end, cursor_end, completed_cycle) with log_extra_metadata_on_done.

With the flag off, the old code path runs unchanged.

A follow-up MR (stacked on this one) handles builds that can never be archived, so they stop coming back on every pass. It also stops builds that are skipped cheaply from counting toward the loop limit. Until then, those skips make a full pass longer.

References

Screenshots or screen recordings

Not applicable: no UI changes.

Database

Plans from gitlab-production-ci (Database Lab), with EXPLAIN (ANALYZE, BUFFERS) on a cold cache. The clone's data ended at about 12:40 UTC on 2026-10-21, so "stale" is finished before 00:37 UTC, and "newest builds" is a cursor at about the first build created then (17333870000). No build above it can be stale yet, which is what the end of every pass looks like.

Before (current query, first batch): 1,507 s. It hashes all (build_id, partition_id) pairs (1.64M) and looks up every candidate build (1.56M) before LIMIT 100.

After, query 1: the build-key window.

SELECT DISTINCT "ci_build_trace_chunks"."build_id", "ci_build_trace_chunks"."partition_id"
FROM "ci_build_trace_chunks"
WHERE "ci_build_trace_chunks"."build_id" > :cursor
  AND "ci_build_trace_chunks"."build_id" <= :pass_end
ORDER BY "ci_build_trace_chunks"."build_id" ASC, "ci_build_trace_chunks"."partition_id" ASC
LIMIT 100
Cursor Plan Cold Buffers Rows
0 (start of a pass) plan 758 ms 1,429 100
newest builds plan 638 ms 887 100

The plan is an Index Scan on index_ci_build_trace_chunks_on_build_id_and_chunk_index with Index Cond: ((build_id > cursor) AND (build_id <= pass_end)), then Incremental Sort and Unique, stopping at 100. Cold time is almost all disk reads.

After, query 2: load the stale builds in the window (plan).

SELECT "p_ci_builds".* FROM "p_ci_builds"
WHERE "p_ci_builds"."type" = 'Ci::Build'
  AND "p_ci_builds"."id" IN (:up_to_100_ids)
  AND "p_ci_builds"."partition_id" IN (:partition_ids)
  AND "p_ci_builds"."status" IN ('success', 'failed', 'canceled')
  AND (finished_at < :twelve_hours_ago)
ORDER BY "p_ci_builds"."id" ASC
  • 624 ms cold, 1,531 buffers, 77 rows. Each partition is checked with a primary-key index scan; there are no sequential scans.
  • EXPLAIN can't return production ids, so on the clone the ids and partitions were passed as = ANY(ARRAY(<window query>)) in place of the literal lists. With literal lists, PostgreSQL also skips unused partitions at plan time.

After, query 3: the pass end, once at the start of each pass. A backward scan of the same index: 1.9 ms cold, 6 buffers.

SELECT MAX("ci_build_trace_chunks"."build_id") FROM "ci_build_trace_chunks"

A run of about 20 batches spends seconds in SQL.

Why not one query that joins the chunks to the builds? It's closer to the old shape, but its LIMIT 100 only stops after 100 stale builds. Near the end of a pass only a few are left, so it reads every newer chunk and looks up each build to find them:

  • 12 rows a little below the newest builds, after reading 257,969 chunk rows (plan).
  • 0 rows at the newest builds, after 257,445 chunk rows and 110,672 build lookups: 1.55 min cold (plan).

That happens at the end of every pass, against a 15 s statement timeout. The window query does a fixed amount of work at any cursor.

Why the pass end: build IDs on GitLab.com rose by about 1.22M an hour (about 340 a second) going by public job IDs, and one window takes about 12 ms. Near the newest builds, a window almost never comes back empty, so without the bound a pass would keep reading new builds until the time limit, every run.

Tradeoff: builds the cursor passes before they're stale (still running, or finished less than 12 hours ago), and builds created after a pass starts, are only looked at on the next pass. A skipped build is saved only if the next pass reaches it before its trace chunks expire from Redis (CHUNK_REDIS_TTL, 7 days). While the existing backlog drains, passes are much longer than that. With this MR alone, builds that can't be archived also count toward the loop limit, so passes stay long. The follow-up MR stops that, and draining the backlog is out of scope here.

Known limit: builds with many chunks (long logs) make their window read more index rows, one per chunk. On GitLab.com chunks average under 3 per build.

How to set up and validate locally

  1. Enable incremental logging and the flag in the Rails console. GDK without object storage rejects the setting through validation, so set it directly:

    s = ApplicationSetting.current_without_cache
s.assign_attributes(ci_job_live_trace_enabled: true)
s.save!(validate: false)
ApplicationSetting.expire
Feature.enable(:ci_archive_traces_cron_cursor)

  2. Give a few finished builds a live trace and make them stale:

    builds = Ci::Build.finished.without_archived_trace.last(5)
builds.each do |b|
  b.update_columns(finished_at: 1.day.ago)
  Gitlab::ExclusiveLease.skipping_transaction_check { b.trace.send(:unsafe_set, "log for #{b.id}\n" * 100) }
end

  3. Run the cron with a small loop limit to see the cursor carry over between runs:

    Ci::ArchiveTraceService.send(:remove_const, :LOOP_LIMIT); Ci::ArchiveTraceService.const_set(:LOOP_LIMIT, 2)
3.times { w = Ci::ArchiveTracesCronWorker.new; w.perform; pp w.instance_variable_get(:@done_log_extra_metadata) }

    The first runs stop with limit_reached: true and pass their cursor_end to the next run, all with the same pass_end. The last run reports completed_cycle: true.

  4. Check that the traces are archived and the chunks are gone:

    builds.map { |b| [b.reload.job_artifacts_trace.present?, b.trace_chunks.count] }

I also ran this against about 280 seeded builds (multi-chunk logs, running, recently finished and archived-pipeline builds), adding stale builds above pass_end partway through the pass. Every stale log was archived byte for byte, the builds added mid-pass were left for the next pass and archived there, and the other builds were left alone.

Example run stats from GDK (280 seeded builds: 260 stale, 10 finished less than 12 hours ago, 5 running, 5 in archived pipelines), from the Ci::ArchiveTracesCronWorker done line in log/sidekiq.log:

{
  "class": "Ci::ArchiveTracesCronWorker",
  "message": "Ci::ArchiveTracesCronWorker JID-991abe4b21baf50f20890e63: done: 6.403865 sec",
  "job_status": "done",
  "extra.ci_archive_traces_cron_worker.result": {
    "processed": 260,
    "skipped_archived_pipeline": 5,
    "limit_reached": false,
    "over_time": false,
    "cursor_start": 0,
    "pass_end": 9732,
    "completed_cycle": true,
    "cursor_end": null
  },
  "duration_s": 6.403865
}

All 260 stale builds were archived in one pass. The recently finished and running builds were left for a later pass, and the cursor was cleared at the end.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Max Fan

Merge request reports