Archive what survives of traces out of archival attempts

Max Fanrequested to merge
634314-archive-incomplete-traces into master

What does this MR do and why?

Builds whose trace can't be archived stay candidates for Ci::ArchiveTracesCronWorker forever. A trace chunk in Redis expires after a week. If a middle chunk is lost, every archival attempt fails until the attempts run out. After that, attempt_archive_cleanup! only acts when an archive exists, so the chunk rows stay, and every cron pass picks the build up again.

This MR gives those builds an end state. It ships behind ci_archive_traces_cron_cursor, the same flag as !263376 (merged), so the cron and its end states roll out together.

  • End state for builds out of attempts. When the regular attempts run out, the new Gitlab::Ci::Trace#archive_incomplete! archives the chunks that survive, in order, with a marker line in place of each lost one, then deletes the chunks. The first try runs as soon as the attempts run out, which is 6–6.8 days in, before the 7-day Redis expiry (see Gitlab::Ci::Trace::Backoff). If it fails, it's retried every 6 hours until it succeeds, so a brief failure still gets another try before the expiry. A trace whose data is all gone is saved as a markers-only archive. Nothing is deleted without being archived. The service logs Archived incomplete trace. with the missing chunk indexes.
  • Tell lost data from empty data. The new Ci::BuildTraceChunk#stored_data returns nil when the chunk's store has nothing. #data turns that into "". Incomplete archival only runs once the regular attempts run out, about 6 days after the build finished. By then no flush can still be running (Ci::BuildTraceChunkFlushWorker doesn't retry), so a chunk that reads as empty has lost its data.
  • Observability. execute now returns its outcome: archived, incomplete, failed, nothing_to_archive, skipped_backoff, skipped_archived_pipeline or already_archived. Each outcome is counted in the new gitlab_ci_trace_archival_total{result, worker} counter and in the cron's run stats.
  • Loop limit. Only builds where archival ran (archived, incomplete, failed) count toward LOOP_LIMIT. Skips cost little and are bounded by the run's time limit. Otherwise a backlog of skipped builds could stretch one pass past the 7-day Redis TTL of the chunks still waiting.

archive! itself doesn't change. A trace whose last chunk expired is still archived shortened, with no error, as today. That doesn't leave builds stuck, so detecting it is left for #635298.

With the flag off, behaviour is unchanged: execute calls legacy_execute, which is the previous execute unchanged. It's removed with the flag.

References

Screenshots or screen recordings

Not applicable: no UI changes.

Database

There are no new batch queries. The one new query reads one build's chunk rows, without raw_data, when a build out of attempts is archived incomplete:

Query and plan 
SELECT "ci_build_trace_chunks"."id", "ci_build_trace_chunks"."chunk_index", "ci_build_trace_chunks"."data_store",
  "ci_build_trace_chunks"."checksum", "ci_build_trace_chunks"."lock_version", "ci_build_trace_chunks"."build_id",
  "ci_build_trace_chunks"."partition_id", "ci_build_trace_chunks"."project_id"
FROM "ci_build_trace_chunks"
WHERE "ci_build_trace_chunks"."build_id" = :build_id AND "ci_build_trace_chunks"."partition_id" = :partition_id
ORDER BY "ci_build_trace_chunks"."chunk_index" ASC

1.2 ms, 16 buffers on gitlab-production-ci: Index Scan on the unique index_ci_build_trace_chunks_on_build_id_and_chunk_index. It's the existing trace_chunks association, minus one column, and only runs for builds out of attempts.

  • Each chunk is then read one at a time, so only one chunk is in memory. Chunks stored in the database are loaded by primary key when read.
  • Deleting the chunks is the existing fast_destroy_all.

How to set up and validate locally

  1. Enable incremental logging and the flag in the Rails console:

    s = ApplicationSetting.current_without_cache
s.assign_attributes(ci_job_live_trace_enabled: true)
s.save!(validate: false) # GDK without object storage fails this validation
ApplicationSetting.expire
Feature.enable(:ci_archive_traces_cron_cursor)

  2. Seed a stale build with a 3-chunk trace that is out of archival attempts, then expire its middle chunk in Redis:

    b = Ci::Build.finished.without_archived_trace.last
b.update_columns(finished_at: 1.day.ago)
data = 'a' * 128.kilobytes + 'b' * 128.kilobytes + 'c' * 10
Gitlab::ExclusiveLease.skipping_transaction_check { b.trace.send(:unsafe_set, data) }
Gitlab::Redis::TraceChunks.with { |r| r.del("gitlab:ci:trace:#{b.id}:chunks:1") }
Ci::BuildTraceMetadata.find_or_upsert_for!(b.id, b.partition_id, b.project_id)
  .update_columns(archival_attempts: 6, last_archival_attempt_at: 1.week.ago)

  3. Run the cron and look at the archive:

    w = Ci::ArchiveTracesCronWorker.new; w.perform; pp w.instance_variable_get(:@done_log_extra_metadata)
b.reload.job_artifacts_trace.file.read.then { |log| [log.count('a'), log[/\[GitLab:.*\]/], log.count('c')] }

    You should see results: { incomplete: 1 }, then [131072, "[GitLab: part of this job log was lost (chunk 1)]", 10].

I also ran this against about 80 seeded builds covering lost last, middle and only chunks, builds out of attempts (just ran out, retrying daily, all data lost, in backoff), and builds that must be left alone. Every build reached its expected end state, archives matched their expected content byte for byte, and no Redis keys were left behind.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Max Fan

Merge request reports