Add observability settings application setting
What does this MR do and why?
- Adds a new checkbox to enable/disable the periodic instance-level job to collect Observability errors from the cloud-connected GitLab Observability Backend.
- Stops those network calls from happening when disabled to avoid network errors piling up in the sidekiq logs.
References
- Customer bug report: Observability::AlertQueryWorker causing "Failed... (#496661 - closed) • Max Woolf • 17.8
MR acceptance checklist
Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.
Screenshots or screen recordings
Screen recording of reproduction: https://gitlab.zoom.us/rec/share/5xrFvI3gHjxIJBjd9DwvlzmcoEudAeJ19hM2LIpyPZCJXMELFHON---bZIMOgkfk.n3RCLedETcTceUf3?startTime=1734600834000
How to set up and validate locally
- Enable the
observability_featuresfeature flag. - Login as an instance admin, and visit
Admin Area -> Network -> GitLab Observability Backend requests. - Try to toggle the new
Fetch cloud observability alertscheckbox, and ensure the result is persisted to the database. - Toggle the flag off.
- Block your computer's access to
observe.staging.gitlab.com. You could add something like182.1.1.34 observe.staging.gitlab.comto/etc/hostswhere the IP is some unreachable or made-up IP address. - Run the worker locally, it should timeout with
Net::OpenTimeout. - Toggle the admin flag on.
- Run the same worker, and it should exit immediately after the licence check.
Related to #496661 (closed)
Edited by Max Woolf
