Add pg_ash sampler thread and admin settings

This MR adds the pg_ash sampling driver, building on the pg_ash installer merged in Phase 0 (!249512 (merged)).

Summary

• Extracts pg_ash schema suppression into a shared Gitlab::Database::PgAsh.execute() helper for reuse
• Adds pg_ash_sampling_enabled and pg_ash_sample_interval_seconds settings (stored in existing database_settings JSONB column); no migration needed
• Adds admin UI (Settings > Metrics and profiling > Active session history) and REST API exposure for both settings
• Implements Gitlab::Database::PgAsh::Sampler with fleet-wide exclusive lease and background thread runner in Sidekiq (PgAshSampler); exports Prometheus metrics and error handling (3-retry limit per sample)
• Updates documentation and regenerates the OpenAPI spec
• Sampling is off-by-default (setting default + separate pg_ash install requirement); no feature flag added, since a flag would be a third on/off switch for the same thing

Deferred to a follow-up MR: rollup and rotation cron jobs (ash.rollup_minute(), ash.rollup_hour(), ash.rollup_cleanup(), ash.rotate()). Without them, ash.sample grows unbounded until an administrator manually uninstalls pg_ash — this is called out in the docs.

References

• Work item: #608100
• Related epic: &23091
• Phase 0 merge request: !249512 (merged)

Testing guide for reviewers

Prerequisites

1. Confirm GDK is running:

   gdk status

   Expected: postgresql and rails-background-jobs are up.

2. Check out the branch:

   git checkout 608100-pg-ash-sampler

1. Install pg_ash

bundle exec rake gitlab:db:pg_ash:install
bundle exec rake gitlab:db:pg_ash:status

Expected: install prints "pg_ash 2.0-beta1 installed." (or "re-applied." on a re-run). Status prints a metric table. Re-running install is safe.

2. Enable sampling

Go to Admin area > Settings > Metrics and profiling. Expand "Active session history". Check "Turn on session sampling". Leave the interval at 1. Save.

Note: the setting reaches the Sidekiq process through a per-process cache (up to 60 s) plus a 30 s lease-retry tick. Sampling can start up to ~90 s after saving. No restart needed.

3. Verify sampling happens

ASH only records backends active at the sampling instant. An idle GDK produces no rows; a flat count does not mean sampling is broken. Hold a query active to check:

psql -h <gdk-root>/postgresql -d gitlabhq_development -c "select pg_sleep(10)"
psql -h <gdk-root>/postgresql -d gitlabhq_development -c "select count(*) from ash.sample"

Expected: the count grows by ~10 (one sample per second of active query).

4. Verify single sampler

The lease key is gitlab:exclusive_lease:pg_ash_sampler:main:lock in shared-state Redis. Only one process should hold it. Check log/application_json.log for other processes; they log "Cannot obtain an exclusive lease" at INFO.

5. Change the sampling interval

In the same admin form, set the interval to 5 and save. Wait ~90 s. Repeat the check with a longer sleep:

psql -h <gdk-root>/postgresql -d gitlabhq_development -c "select pg_sleep(20)"
psql -h <gdk-root>/postgresql -d gitlabhq_development -c "select count(*) from ash.sample"

Expected: the count grows by ~4 (one sample per 5 s). No restart needed.

6. Disable sampling

Uncheck "Turn on session sampling" and save. Wait ~90 s.

psql -h <gdk-root>/postgresql -d gitlabhq_development -c "select pg_sleep(10)"
psql -h <gdk-root>/postgresql -d gitlabhq_development -c "select count(*) from ash.sample"

Expected: no new rows. The Redis lease key is gone. pg_ash's own config reflects the change:

select sampling_enabled from ash.config where singleton;  -- returns f

7. Health checks

At any point, confirm no errors:

select missed_samples, insert_errors, skipped_samples from ash.config where singleton;

Expected: all three stay 0. bundle exec rake gitlab:db:pg_ash:status shows the same values via ash.status().

8. Optional cleanup

bundle exec rake gitlab:db:pg_ash:uninstall

Drops the ash schema and all samples. If the setting is still enabled after uninstall, the sampler is a silent no-op (schema-presence guard).