Extract tenant-observability-stack into its own module
The "observability stack" in Cells is based on what infrastructure Instrumentor deploys. This uses the tenant-observability-stack module in here.
We would like to extract this module into its own codebase and have Instrumentor use it, instead of keeping this inside Instrumentor.
This increases isolation, which helps us to speed up iteration cycles and also forces us to maintain a clean and well defined interface for this module.
@andrewn explained that it'll be beneficial to extract well defined parts of the Instrumentor codebase, so that teams can work more independently (among other reasons). These modules should all follow a common and well defined structure, which allows us to hook into Instrumentor in the right places at the right time (think Instrumentor phases: onboard,prepare,provision etc.).
Definition of Done
-
tenant-observability-stack
module extracted into separate project -
Project has baseline CI including linting and publishes a TF module -
Project has mkdocs documentation set up and deployed to Pages -
Module development happens on the GitLab.com instance (not ops) -
Docs have instructions on how to test module locally and outside of Instrumentor -
Integration into Instrumentor to use the module in GCP -
Cleanup and remove https://gitlab.com/gitlab-com/gl-infra/terraform-modules/observability/observability-cell-stack
Status 2024-06-13
- Project for module created: https://gitlab.com/gitlab-com/gl-infra/terraform-modules/observability/tenant-observability-stack
- Implemented batch job to update Prometheus rules instead of using a helm chart
- Integration back into Instrumentor: https://gitlab.com/gitlab-com/gl-infra/gitlab-dedicated/instrumentor/-/merge_requests/3320 (wip)