Skip to content

Enable multiple queries per Prometheus graph

Mike Greiling requested to merge multiple-query-prometheus-graphs into master

What does this MR do?

Enables multiple queries to be rendered in a single graph (part of gitlab-org/gitlab-ee#2594)

New Behavior

  • when multiple "queries" exist within a "metric", they are all rendered in the same graph (previously all queries after the first were ignored)
  • the second query (if it exists) will have its results rendered with dashed lines, the third (if it exists) will be represented with dotted lines.
  • the "track" property in the query will be used to label the query within the legend
  • currently it is assumed that multiple queries will all use the same units and scale (comparing, say, MB of memory used versus % of CPU consumption in a single graph will not work as expected, since one is from 0.0 to 1.0 and the other may be 0 to 1024)

To test a second query within a graph, you must add another query to an existing metric in config/prometheus/additional_metrics.yml and include a track property to label the results from this query:

- group: System metrics (Kubernetes)
  priority: 5
  metrics:
  - title: "Memory Usage"
    y_label: "Memory Usage (MB)"
    required_metrics:
      - container_memory_usage_bytes
    weight: 1
    queries:
    - query_range: '(sum(container_memory_usage_bytes{container_name!="POD",environment="%{ci_environment_slug}"}) / count(container_memory_usage_bytes{container_name!="POD",environment="%{ci_environment_slug}"})) /1024/1024'
      label: Average
      unit: MB
    - query_range: '(sum(container_memory_usage_bytes{container_name!="POD",environment="%{ci_environment_slug}-canary"}) / count(container_memory_usage_bytes{container_name!="POD",environment="%{ci_environment_slug}-canary"})) /1024/1024'
      label: Average
      unit: MB
      track: canary
  - title: "CPU Utilization"
    y_label: "CPU Utilization (%)"
    required_metrics:
     - container_cpu_usage_seconds_total
    weight: 1
    queries:
    - query_range: 'sum(rate(container_cpu_usage_seconds_total{container_name!="POD",environment="%{ci_environment_slug}"}[2m])) * 100'
      label: CPU
      unit: "%"
    - query_range: 'sum(rate(container_cpu_usage_seconds_total{container_name!="POD",environment="%{ci_environment_slug}-canary"}[2m])) * 100'
      label: CPU
      unit: "%"
      track: canary

Screenshots (if relevant)

Screen_Shot_2017-11-05_at_11.28.38_PM

Does this MR meet the acceptance criteria?

What are the relevant issue numbers?

gitlab-org/gitlab-ee#2594

Edited by Mike Greiling

Merge request reports