Fix empty partition range crashing pipeline lookups (CI outage)
What does this MR do and why?
Partition cache lookups crashed when a CI partition held exactly one pipeline. Ci::Partitions::SyncService wrote exclusive ranges (min...max) into ci_partitions.pipelines_id_range (an int8range column). When min == max, the range was empty, so PostgreSQL stored it as 'empty'. PartitionCache#load_from_database only filtered SQL NULL, so the empty row loaded. ActiveRecord cast the empty int8range to nil, and PartitionCache#store then called nil.begin, raising NoMethodError.
This happens inside Ci::Pipeline.find_by_id, so every Sidekiq worker that looks up a pipeline crashed. New pipelines stayed in created and jobs were never dispatched to runners, causing an instance-wide CI outage.
The exclusive upper bound also dropped the pipeline whose id equals max into a gap between the outgoing and incoming partition ranges.
Changes
app/services/ci/partitions/sync_service.rb: use inclusive ranges (min..maxfor the outgoing partition,(max.next..)for the incoming one) so a single-pipeline range is never empty and themaxpipeline is covered.lib/gitlab/ci/pipeline/partition_cache.rb: skip empty ranges when loading from the database withNOT isempty(pipelines_id_range), so an already-persisted empty range can no longer crash lookups.
Database
PartitionCache#load_from_database gains the NOT isempty(...) filter. The generated query is:
SELECT "ci_partitions"."id", "ci_partitions"."pipelines_id_range"
FROM "ci_partitions"
WHERE "ci_partitions"."pipelines_id_range" IS NOT NULL
AND (NOT isempty(pipelines_id_range))ci_partitions has a handful of rows, so this is a trivial sequential scan.
Query plan: https://postgres.ai/console/gitlab/gitlab-production-ci/sessions/56446/commands/160664
Time: 10.247 ms
- planning: 0.509 ms
- execution: 9.738 ms
- I/O read: 9.615 ms
- I/O write: 0.000 ms
Shared buffers:
- hits: 0 from the buffer pool
- reads: 2 (~16.00 KiB) from the OS file cache, including disk I/O
- dirtied: 0
- writes: 0When was this introduced
The read path that triggers the crash first shipped in 19.1. The empty range could be written earlier, but nothing read it through PartitionCache until then.
| Commit | Date | First stable branch | Role |
|---|---|---|---|
dbfb6ee64a91 remove feature flag ci_time_based_partitioning |
2026-03-02 | 19-0 | write-switching on by default |
32407647302d track pipelines_id_range |
2026-05-12 | 19-0 | sync service starts writing the exclusive range |
b040dc6d0b43 resolve Ci::Pipeline by id via pipelines_id_range |
2026-05-22 | 19-1 | wires find_by_id to ByIdLookup and PartitionCache, the crash path |
find_by_id calls ByIdLookup unconditionally (app/models/ci/pipeline.rb); there is no feature flag gating it. So the empty-range write has existed since 19.0, but nothing read the range through PartitionCache until 19.1.
The report cites 19.2.1 because writing an empty range needs a real partition switch where the closing partition holds exactly one pipeline (min == max). Switches only fire once an instance's ci_partitions_in_seconds_limit window elapses, so this is a data-and-timing condition, not a version gate. This customer's first qualifying switch landed on a single-pipeline boundary while they were on 19.2.1; the vulnerable code was present from 19.1 on.
Testing
spec/services/ci/partitions/sync_service_spec.rb and spec/lib/gitlab/ci/pipeline/partition_cache_spec.rb: 25 examples, 0 failures. New cases cover a single-pipeline partition and an empty range in the cache.
For instances already affected, the issue workaround still applies: fix the bad row's range and call Gitlab::Ci::Pipeline::PartitionCache.invalidate.
Closes #608165 (closed)