Roll up and rotate pg_ash samples from Sidekiq cron

Summary

• Add a cron worker to run pg_ash rollup and retention on a schedule. Without it, raw samples in ash.sample grow without bound.
• Database::PgAshMaintenanceWorker runs four pg_ash functions on the primary, through Gitlab::Database::PgAsh::Maintenance#run:
  • rollup_minute — every minute, rolls raw samples into 1-minute buckets
  • rollup_hour — hourly, rolls 1-minute buckets into 1-hour buckets
  • rotate — hourly from 00:03 to 05:03, advances the raw partition ring and truncates the oldest slot once a day; the other runs retry a failed rotation or return skipped: rotated too recently
  • rollup_cleanup — hourly from 03:07 to 05:07, deletes rollup rows past the configured retention
• rotate rolls up the slot it is about to truncate, then fails if a minute inside rollup_1m_retention_days is still missing from ash.rollup_1m, and counts consecutive failures in ash.config. rotate and rollup_minute compete for the same advisory lock with a try-lock, so a lost race would cost a day of rotation without the hourly retry window.
• Add a Prometheus gauge, gitlab_pg_ash_consecutive_rotate_failures, to the existing sampler metrics.
• Gate: same as the sampler. Runs only when pg_ash_sampling_enabled is on, the database is not read-only, and the ash schema is installed.
• Retention values (rotation_period, num_partitions, rollup_1m_retention_days, rollup_1h_retention_days) keep the pg_ash defaults in ash.config: raw history is one full day plus the current day, minute rows 30 days, hour rows 5 years. Admin settings for them are a follow-up in the work item.
• Update the docs to describe rollup and retention, replacing the note that samples grow until uninstall.

References

• Work item: #608100
• Epic: gitlab-org#23091
• Follows !250170 (merged) (sampler thread and admin settings)
• Follows !249512 (merged) (Phase 0 installer)

Testing guide for reviewers

Steps for GDK:

1. Install and enable sampling: bundle exec rake gitlab:db:pg_ash:install, then turn on Active session history in Admin > Settings > Metrics and profiling (see the testing guide in !250170 (merged)).

2. Generate activity so a full minute has samples:

   psql -h <gdk-root>/postgresql -d gitlabhq_development -c "select pg_sleep(70)"

3. Run the minute rollup by hand instead of waiting for cron:

   bundle exec rails runner 'p Database::PgAshMaintenanceWorker.new.perform("rollup_minute")'
   psql -h <gdk-root>/postgresql -d gitlabhq_development -c "select count(*) from ash.rollup_1m"

   Expect the runner to print the number of minutes rolled up, and a count above 0. Sidekiq cron also runs this every minute; check log/sidekiq.log for PgAshMaintenanceWorker with a result field.

4. Repeat for the other tasks:

   • rollup_hour — needs a completed hour, otherwise prints "0".
   • rotate — prints skipped: rotated too recently at ... unless a day has passed since install. That is expected.
   • rollup_cleanup — prints cleanup: deleted 0 minute rows, 0 hourly rows on a fresh install.

5. Run bundle exec rake gitlab:db:pg_ash:status and check last_rollup_1m_ts, rotated_at, and consecutive_rotate_failures.

6. Turn sampling off, then run a task by hand again. Expect the runner to print nil: the worker returns before any SQL runs.

🤖 Generated with Claude Code