Set up ops staging environment
We currently do not have a way to test Thanos query as the only deployment is in ops
and is used as production.
To ensure we are able to test any update effectively, we propose to have a separate environment that is a replica of the one in ops
.
To do this we have explored 2 options:
- Deploy the same
ops
deployment in a new namespace in theops
cluster-
Spec:
- Create a new namespace e.g.
monitoring-stg
- Create domains and static IPs
- Deploy Thanos query
- Create a new namespace e.g.
-
Pros:
- Exactly the same setup as a production i.e. linked to the same Thanos store
-
Cons:
- Creating multiple namespaces for the environment
-
Spec:
- Have a separate environment in
gstg
though with a drawback as we will not be able to get all the metrics-
Spec:
- Deploy query into the existing namespace
- Create domains and static IPs
-
Cons:
- It's not a replica of production as staging has its own Thanos store
-
Spec:
Merge Requests
-
Create Thanos Query DNS 👉 https://ops.gitlab.net/gitlab-com/gl-infra/config-mgmt/-/merge_requests/3560 -
Add IAM identity 👉 https://ops.gitlab.net/gitlab-com/gl-infra/config-mgmt/-/merge_requests/3587 -
Create the thanos-iap-oauth
secretkubectl create namespace monitoring-stg kubectl --namespace monitoring-stg create secret generic thanos-iap-oauth kubectl --namespace monitoring-stg edit secret thanos-iap-oauth
-
Setup Thanos Query environment 👉 gitlab-com/gl-infra/k8s-workloads/tanka-deployments!343 (merged) -
Incident management issue 👉 production#6637 (closed)
Edited by Silvester Wainaina