ci(integration): faster test postgres and a migrations job split

Summary

The three test:integration jobs (PostgreSQL 16/17/18 matrix) have been running close to their 25m timeout: recent successful runs on main landed at 666–1322s (16), 580–1228s (17), and 1136–1447s (18), against the 1500s cap — and the cap has already been hit: test:integration: [POSTGRES, 18] on main failed with job_execution_timeout at 1503s on 2026-08-13. The job's own traces put internal/datastore/migrations at 771s of the ~1404s test phase — that package replays the 79-migration chain repeatedly (Up/Down/Up, the concurrent-instance test, each schema suite's DownTo walk), serially. This MR removes the durability overhead from those DDL-bound passes, splits the critical-path package into its own job, and makes the job's tooling and triggers more deterministic.

Changes

1. Run the test postgres services with durability off

Adds fsync=off, synchronous_commit=off, full_page_writes=off, and autovacuum=off to the shared .pg-service-options anchor, alongside the existing lock-budget and shared-memory options.

  • Every database these jobs create is dropped when the job ends, so write durability is pure overhead on a DDL-bound workload. Nothing the suites assert depends on crash durability or vacuum timing.
  • maintenance_work_mem deliberately stays at its default (addressed in review): it is a per-backend cap, several packages run migration chains concurrently against one service, and the chain's index builds run against empty partitions — raising it would add aggregate memory pressure on constrained runners for no measurable speedup.
  • Consumers: test:integration (×3), test:integration:migrations (×3, see change 4), test:integration:conformance:s3-garage, test:integration:conformance:gcs-key-creds, and the OCI/npm/Maven conformance jobs via .conformance-ar-env.
  • db:migrate, db:structure-check, and jet:generate-check keep their bare postgres services with every production-default GUC: the lock-budget parity gate (max_locks_per_transaction=64) is untouched (verified: db:migrate durations unchanged at 318–406s across pipelines).

2. Pin gocover-cobertura to v1.5.0

Replaces @latest in test:integration, the Redis topology jobs, and test:integration:s3-garage with the version the real-backend jobs already pin, so an upstream release cannot recompile out from under the go-build cache or change the coverage conversion underneath a green pipeline.

3. Rerun test:integration on .gitlab-ci.yml changes

Adds .gitlab-ci.yml to the &integration-test-files changes anchor, matching the sibling integration jobs whose changes lists already carry this path. Without it, an edit to the job definition (like the ones in this MR) first runs the job on main.

4. Split internal/datastore/migrations into test:integration:migrations

The migrations suite is serial by construction (tests share one database and walk the chain up and down) and sat on the critical path: its window ended within 1s of the gotestsum DONE line while every other package had finished. The new job runs just that package on the same PG matrix, Postgres-only (no Garage/Redis services — the suite exercises the schema alone), with its own 30m timeout budget. test:integration keeps the remaining packages; each uploads its own cobertura report, which GitLab merges in the MR view. The two jobs share their versioned postgres service entry, cobertura artifacts block, coverage regex, and rules list via YAML anchors defined on test:integration (&pg-test-service, &cobertura-report, &coverage-regex, &integration-rules), so the shared blocks cannot drift on a future edit.

Measured results

test:integration job durations (pipeline 2760996495, changes 1–4; identical timings expected from pipeline 2761045892 after dropping maintenance_work_mem, since index builds run against empty partitions):

Variant main durability only + split Δ vs main
PG 16 1319s 1171s 690s −48%
PG 17 1160s 1134s 754s −35%
PG 18 1426s 1126s 816s −43%

test:integration:migrations (new): PG16 730s, PG17 812s, PG18 445s — each leg ran the full 1,047-test suite; the spread is runner variance, not coverage.

Critical path: the main job's tail is now internal/datastore (~655s window), not migrations. Both jobs run concurrently (needs: []), so the integration family's wall clock is max(816s, 812s)13.6 min worst-case, vs ~23.8 min on main.

Timeout headroom: test:integration at 54% of its 25m cap (was 80–97%); test:integration:migrations at 45% of its 30m cap.

Total test volume unchanged: 12,618 (main) + 1,047 (migrations) = 13,665 — exactly what main runs in one job.

Validation

  • YAML load of the expanded config: all anchor consumers resolve the service command; db:migrate keeps its bare service; test:integration:migrations resolves matrix + caches and reuses the &integration-test-files anchor.
  • Local go list -tags=integration with the grep -vE '/datastore/(jet|migrations)(/|$)' filter: migrations and jet excluded from the main job; the new job targets exactly ./internal/datastore/migrations.
  • scripts/ci/check-integration-test-wiring.sh: OK. yamllint: one pre-existing warning class, unchanged from main.

Follow-ups (not in this MR)

  • Add retry on stuck_or_timeout_failure and interruptible: true to the long integration jobs.
  • File-wide dedup refactor: the coverage post-steps (go tool cover + pinned gocover-cobertura install + conversion) are duplicated across ~7 jobs/templates, and the postgres test-DSN variable block across ~3 — a scripts/ci/coverage-cobertura.sh (the install-goose.sh precedent) plus shared anchors would cover them. This MR only anchors the blocks shared between the two matrix jobs.
Edited by Moaz Khalifa

Merge request reports

Loading
Loading