## Summary Track process restart frequency using a filesystem-based approach (timestamp files written to disk on each process start). Expose restart counts for 1m/5m/15m windows via both the process health relay (for Rails) and Prometheus `/metrics` endpoint (for Grafana). **Repository**: https://gitlab.com/gitlab-org/gitlab-zoekt-indexer ## Details - On startup, each process (indexer and webserver) writes a timestamped marker file to `<index_dir>/process_starts/<mode>/` - A `GetStartCounts()` function reads these marker files and counts how many fall within each time window (1m, 5m, 15m) - Old marker files are cleaned up periodically (e.g., keep last 60 minutes) - Restart counts are exposed as Prometheus gauges: `zoekt_process_restarts{window="1m|5m|15m", process="indexer|webserver"}` - Restart counts are included in the `process_health` heartbeat payload (indexer) and the webserver-to-indexer health push ## How It Will Be Used On the Rails side, `restarts_15m >= 2` marks a node as unhealthy and removes it from search routing. A single restart is forgiven (transient OOM, deploy, etc.). Recovery is automatic: once the restarts age out of the 15m window, the count drops and the node becomes healthy again. The same metrics exposed via `/metrics` allow monitoring in Grafana for alerting and dashboards. ## Notes - A previous attempt exists on branch `530296-track-process-starts` in the indexer repo — it can be used as reference but we should start fresh - Previous issue: https://gitlab.com/gitlab-org/gitlab/-/issues/530296