Observability: Default Metrics
What
Every Runway service will be provided default application and container metrics without configuration required. Run stackdriver-exporter
with run.googleapis.com
metric types for runway-staging
, and runway-production
GCP projects.
Why
Right now, Cloud Run Metrics UI requires viewer access for every Runway service owner. Default metrics are required to provide service owners with visibility into the performance, availability, and reliability of their service in dashboards.gitlab.net and thanos.gitlab.net.
Tasks
- gstg
- gprd
Designs
- Show closed items
Relates to
Activity
-
Newest first Oldest first
-
Show all activity Show comments only Show history only
- Chance Feick assigned to @cfeick
assigned to @cfeick
- Chance Feick added to epic gitlab-com/gl-infra&1064 (closed)
added to epic gitlab-com/gl-infra&1064 (closed)
- Chance Feick added workflow-infraReady label
added workflow-infraReady label
- Chance Feick mentioned in issue #67 (closed)
mentioned in issue #67 (closed)
- Chance Feick mentioned in issue #74 (closed)
mentioned in issue #74 (closed)
- Chance Feick marked this issue as related to #74 (closed)
marked this issue as related to #74 (closed)
- Chance Feick mentioned in issue #75 (closed)
mentioned in issue #75 (closed)
- Chance Feick changed title from Default Metrics to Observability: Default Metrics
changed title from Default Metrics to Observability: Default Metrics
- Chance Feick mentioned in issue #77 (closed)
mentioned in issue #77 (closed)
- Chance Feick mentioned in merge request ci-tasks!34 (closed)
mentioned in merge request ci-tasks!34 (closed)
- Chance Feick mentioned in epic gitlab-com/gl-infra&1064 (closed)
mentioned in epic gitlab-com/gl-infra&1064 (closed)
- Chance Feick changed the description
Compare with previous version changed the description
- Chance Feick mentioned in merge request gitlab-org/gitlab!127824 (closed)
mentioned in merge request gitlab-org/gitlab!127824 (closed)
- Author Maintainer
Dependency concern on
aiplatform.googleapis.com
from@alejandro
:Wherever we run it, we need to make sure to carry over the
typePrefixes
config - Author Maintainer
@igorwwwwwwwwwwwwwwwwwwww @ggillies After chatting w/ @alejandro earlier, I'd like to propose that we initially run
stackdriver-exporter
w/ existing k8s workloads. Quite literally just copypasta https://gitlab.com/gitlab-com/gl-infra/k8s-workloads/gitlab-helmfiles/-/tree/master/releases/pvs-stackdriver-exporter?ref_type=heads, e.g.runway-stackdriver-exporter
. Seems quicker than Chef.Huge fan of dogfooding, but I don't think this is a good use case for internal Runway service. Here's why:
- Most (all?) Runway services will be public. Metrics endpoint cannot be public, so we'd start customizing Runway to our internal needs instead of Runway users
- By including our own apps, we'd lose a quick "overview" of all Runway services health and would always need to remember excluding internal services, e.g.
stackdriver_some_metric{service_name!~"\/my-internal-app\/some-other-internal-app\/etc-etc\*"}
Having said that, I haven't actually worked w/ existing k8s workloads yet. I could be underestimating effort involved. Seems like we could reuse prior art for
pvs-stackdriver-exporter
though. What do you think? Collapse replies - Owner
@cfeick This sounds reasonable to me.
Once we have our own GKE cluster we can consider moving it over to the runway GCP projects to make the setup more self-contained.
1
- Chance Feick mentioned in merge request gitlab-com/runbooks!6163 (merged)
mentioned in merge request gitlab-com/runbooks!6163 (merged)
- Chance Feick added workflow-infraIn Progress label and removed workflow-infraReady label
added workflow-infraIn Progress label and removed workflow-infraReady label
- Liam McAndrew mentioned in issue gitlab-com/gl-infra/scalability#2089
mentioned in issue gitlab-com/gl-infra/scalability#2089
- Chance Feick mentioned in merge request gitlab-com/gl-infra/k8s-workloads/gitlab-helmfiles!2777 (merged)
mentioned in merge request gitlab-com/gl-infra/k8s-workloads/gitlab-helmfiles!2777 (merged)
- Author Maintainer
Refs:
- https://ops.gitlab.net/gitlab-com/gl-infra/config-mgmt/-/merge_requests/6518
- https://ops.gitlab.net/gitlab-com/gl-infra/config-mgmt/-/merge_requests/6519
- gitlab-com/gl-infra/k8s-workloads/gitlab-helmfiles!2777 (merged)
- gitlab-com/gl-infra/k8s-workloads/gitlab-helmfiles!2778 (merged)
- https://gitlab.com/gitlab-com/gl-infra/platform/runway/provisioner/-/merge_requests/52
Edited by Chance Feick - Chance Feick changed the description
Compare with previous version changed the description
- Chance Feick added workflow-infraUnder Review label and removed workflow-infraIn Progress label
added workflow-infraUnder Review label and removed workflow-infraIn Progress label
- Chance Feick mentioned in merge request gitlab-com/gl-infra/k8s-workloads/gitlab-helmfiles!2791 (merged)
mentioned in merge request gitlab-com/gl-infra/k8s-workloads/gitlab-helmfiles!2791 (merged)
- Chance Feick changed the description
Compare with previous version changed the description
- Chance Feick marked the checklist item gstg as completed
marked the checklist item gstg as completed
- Chance Feick marked the checklist item gprd as completed
marked the checklist item gprd as completed
- Author Maintainer
Confirmed metrics reporting for both
gstg
andgprd
: - Chance Feick closed
closed
- Chance Feick added workflow-infraDone label and removed workflow-infraUnder Review label
added workflow-infraDone label and removed workflow-infraUnder Review label
- Chance Feick mentioned in issue #76 (closed)
mentioned in issue #76 (closed)
- Sam Wiskow mentioned in epic gitlab-com/gl-infra&1112 (closed)
mentioned in epic gitlab-com/gl-infra&1112 (closed)
- Chance Feick added Category:Runway label
added Category:Runway label