ADR: Document OAK monitoring and logging integration patterns

OAK does not define separate sizing guidelines. Existing Omnibus sizing recommendations apply, and the Kubernetes cluster should meet the resource requirements of the advanced components being deployed. For single-node installations, where the Omnibus and the Kubernetes cluster share the same underlying resources, the minimum sizing must account for both sets of resources.

### Monitoring & Logging

Monitoring and logging for OAK deployments are customer-managed through existing infrastructure patterns. No Omnibus-level changes are required.

Existing GitLab configurations and standard tooling provide sufficient flexibility for users to design solutions that fit their infrastructure.

**Monitoring**: Users can leverage existing `prometheus['scrape_configs']` in Omnibus to consolidate metrics from advanced components. Advanced components expose Prometheus-compatible metrics (for example, OpenBao exposes metrics at `/v1/sys/metrics`). In single-node (colocated) deployments, metrics are accessible on localhost. In deployments with external Kubernetes clusters, users can either expose metrics externally (NodePort, LoadBalancer) to scrape via `prometheus['scrape_configs']`, or use Kubernetes-native patterns (PodMonitor, Prometheus Operator).

**Logging**: GitLab already provides logging functionality. Users are responsible for configuring their log aggregation solution (choosing tools like Filebeat, Fluent Bit, ELK, Loki and others) based on their infrastructure needs. Both Omnibus logs and Kubernetes pod logs can be collected using standard shipper/agent patterns (such as DaemonSet for Kubernetes, agents on VMs for Omnibus).

## Beta Implementation Proposal

Based on discussions with the team, the following represents a proposed Beta implementation of OAK. This Beta phase focuses on establishing a minimal viable product that enables single-node customers to adopt advanced components while intentionally introducing some friction to encourage eventual migration to cloud-native architectures.

6. **Security review**: AppSec review of the architecture and implementation.

7. **Beta release**: Release OAK as a beta feature for early adopters.

### Open Questions & Design Decisions Needed for future iterations

These should be addressed through technical discussions with stakeholders and documented in subsequent iterations of this design document.

#### Monitoring and Logging

**Question**: How should customers monitor and collect logs from advanced components running in OAK?

**Current State**: Omnibus provides Prometheus metrics and logging capabilities for core GitLab components. Since both Omnibus and Kubernetes run on the same VM, the existing Omnibus Prometheus instance could potentially scrape metrics from Kubernetes components running on localhost or the pod network.

**Considerations**:

- **Metrics export**: Advanced components running in Kubernetes may expose Prometheus metrics on localhost ports or the pod network. The Omnibus Prometheus instance could be configured to scrape these metrics directly, providing unified metrics collection.

- **Omnibus Prometheus as aggregator**: Leveraging the existing Omnibus Prometheus infrastructure reduces the need for customers to set up separate monitoring solutions.

- **Log aggregation**: Kubernetes components generate logs that need to be collected and aggregated alongside Omnibus logs. Customers may use solutions like ELK, Loki, or other log aggregation tools.

- **Observability gap**: Currently, there is no documented or supported approach for customers to monitor and log OAK components. This creates an observability gap that could impact customer experience and troubleshooting.

**Options**:

- **No monitoring support**: Monitoring and logging are customer responsibility. Document that customers must implement their own monitoring solution for Kubernetes components.

- **Basic monitoring guidance**: Provide documentation on how to configure Omnibus Prometheus to scrape metrics from Kubernetes components on localhost/pod network. This leverages existing infrastructure without requiring new automation.

- **Omnibus Prometheus integration**: Implement automation to configure Omnibus Prometheus to automatically discover and scrape metrics from advanced components. Omnibus would generate Prometheus scrape configurations for each deployed component.

- **Full monitoring integration**: Implement tooling to automatically export metrics and logs from Kubernetes components to Omnibus monitoring infrastructure, including log aggregation.

**Impact**:

- **No support**: Customers must implement their own monitoring, creating operational overhead.

- **Basic guidance**: Reduces friction by leveraging existing Omnibus Prometheus, but requires manual configuration.

- **Omnibus Prometheus integration**: Provides unified metrics collection with minimal customer effort, but requires Omnibus automation to discover and configure scrape targets.

- **Full integration**: Most user-friendly but adds significant complexity and maintenance burden.

**Recommendation for Beta**: During the Beta phase, experiment with monitoring options.

## References

- [Parent Epic: Omnibus Adjacent Kubernetes (OAK) - Operate Implementation](https://gitlab.com/groups/gitlab-com/gl-infra/software-delivery/-/work_items/22)