Swap bigint columns for ci_sources_pipelines
What does this MR do and why?
This is the swap step of converting the ci_sources_pipelines columns id, project_id, and source_project_id from integer to bigint. It swaps the columns with their *_convert_to_bigint counterparts.
It adds three post-deployment migrations, all milestone 19.4, and they must run in this order:
20260831065014_add_tmp_not_null_check_on_ci_sources_pipelines_project_id.rb20260831065015_validate_tmp_not_null_check_on_ci_sources_pipelines_project_id.rb20260831065016_swap_columns_for_ci_sources_pipelines_bigint_conversion.rb
What the swap does
Before the transaction, it recreates the bigint primary key index with the add_bigint_column_indexes helper, in case that index was dropped. This is the same helper the earlier index-duplication migration used, so the name and options match. The call is idempotent and skips creation when the index already exists.
Inside a lock-retried transaction it:
- swaps each integer column with its
*_convert_to_bigintcounterpart - swaps the column defaults
- resets the trigger functions
- swaps the primary key constraint
ci_sources_pipelines_pkey - swaps the indexes
index_ci_sources_pipelines_on_project_idandindex_ci_sources_pipelines_on_source_project_id - exchanges the names of the two
project_idNOT NULL checks
Everything in the transaction is a catalog operation. No table is rewritten and no constraint is validated there.
There are no foreign keys on the converted columns, and no foreign keys from other tables point at this table's id. So there is no foreign key swap.
db/structure.sql is not changed on purpose. On new installations these columns are already bigint, so skip_bigint_migration? returns true and the migrations do nothing.
The project_id NOT NULL check
project_id is nullable at the column level. Its NOT NULL guarantee comes from CONSTRAINT check_5a76e457e6 CHECK ((project_id IS NOT NULL)).
A check constraint stays on the physical column, so a plain swap would leave it guarding the retired integer column, and the cleanup step would drop it with that column. The sharding key would lose its guarantee for good.
The first two migrations make the bigint column match the integer column, the same way the process already handles indexes and foreign keys: check_5a76e457e6_tmp is added on project_id_convert_to_bigint as NOT VALID, then validated. The swap then only exchanges the two constraint names, so check_5a76e457e6 ends up on the new bigint project_id and the retired column carries the temporary name, which the cleanup step drops.
The check is already valid when the swap runs, so the sharding key is never guarded by a NOT VALID constraint. Exchanging two names is its own inverse, so down reverses it with no extra work.
This is a gap in the conversion tooling, not something specific to this table. BigintConverter mirrors only column-level NOT NULL, so 27 pending conversions have the same exposure. Tracked in #623184.
Ordering
The bigint indexes must exist in production before the swap runs. They shipped in !242049 (merged), which merged for 19.3. This swap targets 19.4, so the indexes are already deployed.
The NOT NULL check must also exist and be valid before the swap renames it. Post-deployment migrations run in timestamp order, so the two prep migrations run first in the same deploy.
Database testing
Run on 2a3b70d5, all three databases pass.
| Migration | main | ci | sec |
|---|---|---|---|
065014 add the check |
4.0 s | 4.7 s | 4.6 s |
065015 validate it |
3.1 s | 147.7 s | 4.1 s |
065016 swap |
9.7 s | 10.0 s | 11.1 s |
The swap runs 18 statements on the ci database, all catalog work, the longest 10.1 ms. There is no ADD CONSTRAINT and no VALIDATE among them.
The validation is flagged for exceeding the 100 ms query guideline, at 143.6 s on ci. That is the full table scan, which any VALIDATE CONSTRAINT on a table this size will hit. It holds SHARE UPDATE EXCLUSIVE, so reads and writes are not blocked, and it sits in its own migration ahead of the swap.
The reported database size change for the swap on ci is -5.38 GiB, consistent with the earlier runs on this branch.
Rollback
down was exercised by hand against a local ci database, since the testing pipeline only runs up. The pre-swap state was rebuilt (the three columns as integer, the three *_convert_to_bigint columns, a sync trigger, the three bigint indexes, one row), then all three migrations were run up and down again.
A dump of columns with types, nullability and defaults, every index definition, every constraint with its convalidated flag, triggers, sequence ownership and the row is identical before and after. The bigint primary key index is recreated, and check_5a76e457e6 returns to the integer column.
Related
- Work item: #500027 (closed)
- Index duplication (merged): !242049 (merged)
- Followed by (clean up): !242063
- Supersedes: !242057 (closed)
- Conversion tooling gap: #623184
- Epic: &15621