Fix empty partition range crashing pipeline lookups (CI outage)

What does this MR do and why?

Partition cache lookups crashed when a CI partition held exactly one pipeline. Ci::Partitions::SyncService wrote exclusive ranges (min...max) into ci_partitions.pipelines_id_range (an int8range column). When min == max, the range was empty, so PostgreSQL stored it as 'empty'. PartitionCache#load_from_database only filtered SQL NULL, so the empty row loaded. ActiveRecord cast the empty int8range to nil, and PartitionCache#store then called nil.begin, raising NoMethodError.

This happens inside Ci::Pipeline.find_by_id, so every Sidekiq worker that looks up a pipeline crashed. New pipelines stayed in created and jobs were never dispatched to runners, causing an instance-wide CI outage.

The exclusive upper bound also dropped the pipeline whose id equals max into a gap between the outgoing and incoming partition ranges.

Changes

  • app/services/ci/partitions/sync_service.rb: use inclusive ranges (min..max for the outgoing partition, (max.next..) for the incoming one) so a single-pipeline range is never empty and the max pipeline is covered.
  • lib/gitlab/ci/pipeline/partition_cache.rb: skip empty ranges when loading from the database with NOT isempty(pipelines_id_range), so an already-persisted empty range can no longer crash lookups.

Database

PartitionCache#load_from_database gains the NOT isempty(...) filter. The generated query is:

SELECT "ci_partitions"."id", "ci_partitions"."pipelines_id_range"
FROM "ci_partitions"
WHERE "ci_partitions"."pipelines_id_range" IS NOT NULL
  AND (NOT isempty(pipelines_id_range))

ci_partitions has a handful of rows, so this is a trivial sequential scan.

Query plan: https://postgres.ai/console/gitlab/gitlab-production-ci/sessions/56446/commands/160664

Time: 10.247 ms
  - planning: 0.509 ms
  - execution: 9.738 ms
    - I/O read: 9.615 ms
    - I/O write: 0.000 ms

Shared buffers:
  - hits: 0 from the buffer pool
  - reads: 2 (~16.00 KiB) from the OS file cache, including disk I/O
  - dirtied: 0
  - writes: 0

When was this introduced

The read path that triggers the crash first shipped in 19.1. The empty range could be written earlier, but nothing read it through PartitionCache until then.

Commit Date First stable branch Role
dbfb6ee64a91 remove feature flag ci_time_based_partitioning 2026-03-02 19-0 write-switching on by default
32407647302d track pipelines_id_range 2026-05-12 19-0 sync service starts writing the exclusive range
b040dc6d0b43 resolve Ci::Pipeline by id via pipelines_id_range 2026-05-22 19-1 wires find_by_id to ByIdLookup and PartitionCache, the crash path

find_by_id calls ByIdLookup unconditionally (app/models/ci/pipeline.rb); there is no feature flag gating it. So the empty-range write has existed since 19.0, but nothing read the range through PartitionCache until 19.1.

The report cites 19.2.1 because writing an empty range needs a real partition switch where the closing partition holds exactly one pipeline (min == max). Switches only fire once an instance's ci_partitions_in_seconds_limit window elapses, so this is a data-and-timing condition, not a version gate. This customer's first qualifying switch landed on a single-pipeline boundary while they were on 19.2.1; the vulnerable code was present from 19.1 on.

Testing

spec/services/ci/partitions/sync_service_spec.rb and spec/lib/gitlab/ci/pipeline/partition_cache_spec.rb: 25 examples, 0 failures. New cases cover a single-pipeline partition and an empty range in the cache.

For instances already affected, the issue workaround still applies: fix the bad row's range and call Gitlab::Ci::Pipeline::PartitionCache.invalidate.

Closes #608165 (closed)

Edited by Stan Hu

Merge request reports

Loading
Loading