Skip to content

Add idle gitlab_runner_jobs metric per runner

Pedro Pombeiro (OOO from July 16th till Aug 7th) requested to merge pedropombeiro/25845/idle-runner-jobs-metrics into main

What does this MR do?

This MR ensures that there is always at least one gitlab_runner_jobs metric per runner.

Why was this MR needed?

Because some systems (like Horizontal Pod Autoscaler) count on metrics not disappearing when no jobs are running.

What's the best way to test this MR?

Use the following command to enable metrics, and open http://localhost:9090/metrics on a browser:

gitlab-runner run --listen-address 127.0.0.1:9090

With the new branch logic, a new gitlab_runner_jobs{executor_stage="idle",runner="...",stage="idle",state="idle"} 0 entry should be visible for each configured runner:

image

Metrics in idle state for main branch 
# HELP gitlab_runner_api_request_statuses_total The total number of api requests, partitioned by runner, endpoint and status.
# TYPE gitlab_runner_api_request_statuses_total counter
gitlab_runner_api_request_statuses_total{endpoint="request_job",runner="1-DdQors",status="204"} 1
gitlab_runner_api_request_statuses_total{endpoint="request_job",runner="8S27vskH",status="204"} 1
gitlab_runner_api_request_statuses_total{endpoint="request_job",runner="USPj1gp5",status="204"} 1
gitlab_runner_api_request_statuses_total{endpoint="request_job",runner="cpwhDr7z",status="204"} 1
gitlab_runner_api_request_statuses_total{endpoint="request_job",runner="my7zCavq",status="204"} 1
gitlab_runner_api_request_statuses_total{endpoint="request_job",runner="zqMtkFxL",status="204"} 1
# HELP gitlab_runner_autoscaling_machine_creation_duration_seconds Histogram of machine creation time.
# TYPE gitlab_runner_autoscaling_machine_creation_duration_seconds histogram
gitlab_runner_autoscaling_machine_creation_duration_seconds_bucket{executor="docker+machine",le="30"} 0
gitlab_runner_autoscaling_machine_creation_duration_seconds_bucket{executor="docker+machine",le="37.5"} 0
gitlab_runner_autoscaling_machine_creation_duration_seconds_bucket{executor="docker+machine",le="46.875"} 0
gitlab_runner_autoscaling_machine_creation_duration_seconds_bucket{executor="docker+machine",le="58.59375"} 0
gitlab_runner_autoscaling_machine_creation_duration_seconds_bucket{executor="docker+machine",le="73.2421875"} 0
gitlab_runner_autoscaling_machine_creation_duration_seconds_bucket{executor="docker+machine",le="91.552734375"} 0
gitlab_runner_autoscaling_machine_creation_duration_seconds_bucket{executor="docker+machine",le="114.44091796875"} 0
gitlab_runner_autoscaling_machine_creation_duration_seconds_bucket{executor="docker+machine",le="143.0511474609375"} 0
gitlab_runner_autoscaling_machine_creation_duration_seconds_bucket{executor="docker+machine",le="178.81393432617188"} 0
gitlab_runner_autoscaling_machine_creation_duration_seconds_bucket{executor="docker+machine",le="223.51741790771484"} 0
gitlab_runner_autoscaling_machine_creation_duration_seconds_bucket{executor="docker+machine",le="+Inf"} 0
gitlab_runner_autoscaling_machine_creation_duration_seconds_sum{executor="docker+machine"} 0
gitlab_runner_autoscaling_machine_creation_duration_seconds_count{executor="docker+machine"} 0
gitlab_runner_autoscaling_machine_creation_duration_seconds_bucket{executor="docker-ssh+machine",le="30"} 0
gitlab_runner_autoscaling_machine_creation_duration_seconds_bucket{executor="docker-ssh+machine",le="37.5"} 0
gitlab_runner_autoscaling_machine_creation_duration_seconds_bucket{executor="docker-ssh+machine",le="46.875"} 0
gitlab_runner_autoscaling_machine_creation_duration_seconds_bucket{executor="docker-ssh+machine",le="58.59375"} 0
gitlab_runner_autoscaling_machine_creation_duration_seconds_bucket{executor="docker-ssh+machine",le="73.2421875"} 0
gitlab_runner_autoscaling_machine_creation_duration_seconds_bucket{executor="docker-ssh+machine",le="91.552734375"} 0
gitlab_runner_autoscaling_machine_creation_duration_seconds_bucket{executor="docker-ssh+machine",le="114.44091796875"} 0
gitlab_runner_autoscaling_machine_creation_duration_seconds_bucket{executor="docker-ssh+machine",le="143.0511474609375"} 0
gitlab_runner_autoscaling_machine_creation_duration_seconds_bucket{executor="docker-ssh+machine",le="178.81393432617188"} 0
gitlab_runner_autoscaling_machine_creation_duration_seconds_bucket{executor="docker-ssh+machine",le="223.51741790771484"} 0
gitlab_runner_autoscaling_machine_creation_duration_seconds_bucket{executor="docker-ssh+machine",le="+Inf"} 0
gitlab_runner_autoscaling_machine_creation_duration_seconds_sum{executor="docker-ssh+machine"} 0
gitlab_runner_autoscaling_machine_creation_duration_seconds_count{executor="docker-ssh+machine"} 0
# HELP gitlab_runner_autoscaling_machine_removal_duration_seconds Histogram of machine removal time.
# TYPE gitlab_runner_autoscaling_machine_removal_duration_seconds histogram
gitlab_runner_autoscaling_machine_removal_duration_seconds_bucket{executor="docker+machine",le="1"} 0
gitlab_runner_autoscaling_machine_removal_duration_seconds_bucket{executor="docker+machine",le="3"} 0
gitlab_runner_autoscaling_machine_removal_duration_seconds_bucket{executor="docker+machine",le="5"} 0
gitlab_runner_autoscaling_machine_removal_duration_seconds_bucket{executor="docker+machine",le="10"} 0
gitlab_runner_autoscaling_machine_removal_duration_seconds_bucket{executor="docker+machine",le="30"} 0
gitlab_runner_autoscaling_machine_removal_duration_seconds_bucket{executor="docker+machine",le="50"} 0
gitlab_runner_autoscaling_machine_removal_duration_seconds_bucket{executor="docker+machine",le="60"} 0
gitlab_runner_autoscaling_machine_removal_duration_seconds_bucket{executor="docker+machine",le="80"} 0
gitlab_runner_autoscaling_machine_removal_duration_seconds_bucket{executor="docker+machine",le="90"} 0
gitlab_runner_autoscaling_machine_removal_duration_seconds_bucket{executor="docker+machine",le="120"} 0
gitlab_runner_autoscaling_machine_removal_duration_seconds_bucket{executor="docker+machine",le="+Inf"} 0
gitlab_runner_autoscaling_machine_removal_duration_seconds_sum{executor="docker+machine"} 0
gitlab_runner_autoscaling_machine_removal_duration_seconds_count{executor="docker+machine"} 0
gitlab_runner_autoscaling_machine_removal_duration_seconds_bucket{executor="docker-ssh+machine",le="1"} 0
gitlab_runner_autoscaling_machine_removal_duration_seconds_bucket{executor="docker-ssh+machine",le="3"} 0
gitlab_runner_autoscaling_machine_removal_duration_seconds_bucket{executor="docker-ssh+machine",le="5"} 0
gitlab_runner_autoscaling_machine_removal_duration_seconds_bucket{executor="docker-ssh+machine",le="10"} 0
gitlab_runner_autoscaling_machine_removal_duration_seconds_bucket{executor="docker-ssh+machine",le="30"} 0
gitlab_runner_autoscaling_machine_removal_duration_seconds_bucket{executor="docker-ssh+machine",le="50"} 0
gitlab_runner_autoscaling_machine_removal_duration_seconds_bucket{executor="docker-ssh+machine",le="60"} 0
gitlab_runner_autoscaling_machine_removal_duration_seconds_bucket{executor="docker-ssh+machine",le="80"} 0
gitlab_runner_autoscaling_machine_removal_duration_seconds_bucket{executor="docker-ssh+machine",le="90"} 0
gitlab_runner_autoscaling_machine_removal_duration_seconds_bucket{executor="docker-ssh+machine",le="120"} 0
gitlab_runner_autoscaling_machine_removal_duration_seconds_bucket{executor="docker-ssh+machine",le="+Inf"} 0
gitlab_runner_autoscaling_machine_removal_duration_seconds_sum{executor="docker-ssh+machine"} 0
gitlab_runner_autoscaling_machine_removal_duration_seconds_count{executor="docker-ssh+machine"} 0
# HELP gitlab_runner_autoscaling_machine_states The current number of machines per state in this provider.
# TYPE gitlab_runner_autoscaling_machine_states gauge
gitlab_runner_autoscaling_machine_states{executor="docker+machine",state="acquired"} 0
gitlab_runner_autoscaling_machine_states{executor="docker+machine",state="creating"} 0
gitlab_runner_autoscaling_machine_states{executor="docker+machine",state="idle"} 0
gitlab_runner_autoscaling_machine_states{executor="docker+machine",state="removing"} 0
gitlab_runner_autoscaling_machine_states{executor="docker+machine",state="stuck-on-removing"} 0
gitlab_runner_autoscaling_machine_states{executor="docker+machine",state="used"} 0
gitlab_runner_autoscaling_machine_states{executor="docker-ssh+machine",state="acquired"} 0
gitlab_runner_autoscaling_machine_states{executor="docker-ssh+machine",state="creating"} 0
gitlab_runner_autoscaling_machine_states{executor="docker-ssh+machine",state="idle"} 0
gitlab_runner_autoscaling_machine_states{executor="docker-ssh+machine",state="removing"} 0
gitlab_runner_autoscaling_machine_states{executor="docker-ssh+machine",state="stuck-on-removing"} 0
gitlab_runner_autoscaling_machine_states{executor="docker-ssh+machine",state="used"} 0
# HELP gitlab_runner_autoscaling_machine_stopping_duration_seconds Histogram of machine stopping time.
# TYPE gitlab_runner_autoscaling_machine_stopping_duration_seconds histogram
gitlab_runner_autoscaling_machine_stopping_duration_seconds_bucket{executor="docker+machine",le="1"} 0
gitlab_runner_autoscaling_machine_stopping_duration_seconds_bucket{executor="docker+machine",le="3"} 0
gitlab_runner_autoscaling_machine_stopping_duration_seconds_bucket{executor="docker+machine",le="5"} 0
gitlab_runner_autoscaling_machine_stopping_duration_seconds_bucket{executor="docker+machine",le="10"} 0
gitlab_runner_autoscaling_machine_stopping_duration_seconds_bucket{executor="docker+machine",le="30"} 0
gitlab_runner_autoscaling_machine_stopping_duration_seconds_bucket{executor="docker+machine",le="50"} 0
gitlab_runner_autoscaling_machine_stopping_duration_seconds_bucket{executor="docker+machine",le="60"} 0
gitlab_runner_autoscaling_machine_stopping_duration_seconds_bucket{executor="docker+machine",le="80"} 0
gitlab_runner_autoscaling_machine_stopping_duration_seconds_bucket{executor="docker+machine",le="90"} 0
gitlab_runner_autoscaling_machine_stopping_duration_seconds_bucket{executor="docker+machine",le="120"} 0
gitlab_runner_autoscaling_machine_stopping_duration_seconds_bucket{executor="docker+machine",le="+Inf"} 0
gitlab_runner_autoscaling_machine_stopping_duration_seconds_sum{executor="docker+machine"} 0
gitlab_runner_autoscaling_machine_stopping_duration_seconds_count{executor="docker+machine"} 0
gitlab_runner_autoscaling_machine_stopping_duration_seconds_bucket{executor="docker-ssh+machine",le="1"} 0
gitlab_runner_autoscaling_machine_stopping_duration_seconds_bucket{executor="docker-ssh+machine",le="3"} 0
gitlab_runner_autoscaling_machine_stopping_duration_seconds_bucket{executor="docker-ssh+machine",le="5"} 0
gitlab_runner_autoscaling_machine_stopping_duration_seconds_bucket{executor="docker-ssh+machine",le="10"} 0
gitlab_runner_autoscaling_machine_stopping_duration_seconds_bucket{executor="docker-ssh+machine",le="30"} 0
gitlab_runner_autoscaling_machine_stopping_duration_seconds_bucket{executor="docker-ssh+machine",le="50"} 0
gitlab_runner_autoscaling_machine_stopping_duration_seconds_bucket{executor="docker-ssh+machine",le="60"} 0
gitlab_runner_autoscaling_machine_stopping_duration_seconds_bucket{executor="docker-ssh+machine",le="80"} 0
gitlab_runner_autoscaling_machine_stopping_duration_seconds_bucket{executor="docker-ssh+machine",le="90"} 0
gitlab_runner_autoscaling_machine_stopping_duration_seconds_bucket{executor="docker-ssh+machine",le="120"} 0
gitlab_runner_autoscaling_machine_stopping_duration_seconds_bucket{executor="docker-ssh+machine",le="+Inf"} 0
gitlab_runner_autoscaling_machine_stopping_duration_seconds_sum{executor="docker-ssh+machine"} 0
gitlab_runner_autoscaling_machine_stopping_duration_seconds_count{executor="docker-ssh+machine"} 0
# HELP gitlab_runner_concurrent The current value of concurrent setting
# TYPE gitlab_runner_concurrent gauge
gitlab_runner_concurrent 4
# HELP gitlab_runner_errors_total The number of caught errors.
# TYPE gitlab_runner_errors_total counter
gitlab_runner_errors_total{level="error"} 0
gitlab_runner_errors_total{level="fatal"} 0
gitlab_runner_errors_total{level="panic"} 0
gitlab_runner_errors_total{level="warning"} 5
# HELP gitlab_runner_limit The current value of concurrent setting
# TYPE gitlab_runner_limit gauge
gitlab_runner_limit{runner="1-DdQors"} 0
gitlab_runner_limit{runner="8S27vskH"} 0
gitlab_runner_limit{runner="USPj1gp5"} 0
gitlab_runner_limit{runner="cpwhDr7z"} 0
gitlab_runner_limit{runner="my7zCavq"} 0
gitlab_runner_limit{runner="zqMtkFxL"} 0
# HELP gitlab_runner_request_concurrency The current number of concurrent requests for a new job
# TYPE gitlab_runner_request_concurrency gauge
gitlab_runner_request_concurrency{runner="1-DdQors"} 1
gitlab_runner_request_concurrency{runner="8S27vskH"} 0
gitlab_runner_request_concurrency{runner="USPj1gp5"} 1
gitlab_runner_request_concurrency{runner="cpwhDr7z"} 1
gitlab_runner_request_concurrency{runner="my7zCavq"} 0
gitlab_runner_request_concurrency{runner="zqMtkFxL"} 1
# HELP gitlab_runner_request_concurrency_exceeded_total Counter tracking exceeding of request concurrency
# TYPE gitlab_runner_request_concurrency_exceeded_total counter
gitlab_runner_request_concurrency_exceeded_total{runner="1-DdQors"} 0
gitlab_runner_request_concurrency_exceeded_total{runner="8S27vskH"} 0
gitlab_runner_request_concurrency_exceeded_total{runner="USPj1gp5"} 0
gitlab_runner_request_concurrency_exceeded_total{runner="cpwhDr7z"} 0
gitlab_runner_request_concurrency_exceeded_total{runner="my7zCavq"} 0
gitlab_runner_request_concurrency_exceeded_total{runner="zqMtkFxL"} 0
# HELP gitlab_runner_version_info A metric with a constant '1' value labeled by different build stats fields.
# TYPE gitlab_runner_version_info gauge
gitlab_runner_version_info{architecture="amd64",branch="HEAD",built_at="unknown",go_version="go1.13.8",name="gitlab-runner",os="darwin",revision="HEAD",version="development version"} 1
# HELP go_gc_duration_seconds A summary of the GC invocation durations.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 2.6993e-05
go_gc_duration_seconds{quantile="0.25"} 3.3127e-05
go_gc_duration_seconds{quantile="0.5"} 6.2563e-05
go_gc_duration_seconds{quantile="0.75"} 9.043e-05
go_gc_duration_seconds{quantile="1"} 0.000105271
go_gc_duration_seconds_sum 0.000683319
go_gc_duration_seconds_count 11
# HELP go_goroutines Number of goroutines that currently exist.
# TYPE go_goroutines gauge
go_goroutines 40
# HELP go_info Information about the Go environment.
# TYPE go_info gauge
go_info{version="go1.13.8"} 1
# HELP go_memstats_alloc_bytes Number of bytes allocated and still in use.
# TYPE go_memstats_alloc_bytes gauge
go_memstats_alloc_bytes 4.174688e+06
# HELP go_memstats_alloc_bytes_total Total number of bytes allocated, even if freed.
# TYPE go_memstats_alloc_bytes_total counter
go_memstats_alloc_bytes_total 1.2086848e+07
# HELP go_memstats_buck_hash_sys_bytes Number of bytes used by the profiling bucket hash table.
# TYPE go_memstats_buck_hash_sys_bytes gauge
go_memstats_buck_hash_sys_bytes 1.448271e+06
# HELP go_memstats_frees_total Total number of frees.
# TYPE go_memstats_frees_total counter
go_memstats_frees_total 110441
# HELP go_memstats_gc_cpu_fraction The fraction of this program's available CPU time used by the GC since the program started.
# TYPE go_memstats_gc_cpu_fraction gauge
go_memstats_gc_cpu_fraction 0.0006174486524471431
# HELP go_memstats_gc_sys_bytes Number of bytes used for garbage collection system metadata.
# TYPE go_memstats_gc_sys_bytes gauge
go_memstats_gc_sys_bytes 2.38592e+06
# HELP go_memstats_heap_alloc_bytes Number of heap bytes allocated and still in use.
# TYPE go_memstats_heap_alloc_bytes gauge
go_memstats_heap_alloc_bytes 4.174688e+06
# HELP go_memstats_heap_idle_bytes Number of heap bytes waiting to be used.
# TYPE go_memstats_heap_idle_bytes gauge
go_memstats_heap_idle_bytes 5.853184e+07
# HELP go_memstats_heap_inuse_bytes Number of heap bytes that are in use.
# TYPE go_memstats_heap_inuse_bytes gauge
go_memstats_heap_inuse_bytes 7.233536e+06
# HELP go_memstats_heap_objects Number of allocated objects.
# TYPE go_memstats_heap_objects gauge
go_memstats_heap_objects 27469
# HELP go_memstats_heap_released_bytes Number of heap bytes released to OS.
# TYPE go_memstats_heap_released_bytes gauge
go_memstats_heap_released_bytes 5.8015744e+07
# HELP go_memstats_heap_sys_bytes Number of heap bytes obtained from system.
# TYPE go_memstats_heap_sys_bytes gauge
go_memstats_heap_sys_bytes 6.5765376e+07
# HELP go_memstats_last_gc_time_seconds Number of seconds since 1970 of last garbage collection.
# TYPE go_memstats_last_gc_time_seconds gauge
go_memstats_last_gc_time_seconds 1.636731721218909e+09
# HELP go_memstats_lookups_total Total number of pointer lookups.
# TYPE go_memstats_lookups_total counter
go_memstats_lookups_total 0
# HELP go_memstats_mallocs_total Total number of mallocs.
# TYPE go_memstats_mallocs_total counter
go_memstats_mallocs_total 137910
# HELP go_memstats_mcache_inuse_bytes Number of bytes in use by mcache structures.
# TYPE go_memstats_mcache_inuse_bytes gauge
go_memstats_mcache_inuse_bytes 27776
# HELP go_memstats_mcache_sys_bytes Number of bytes used for mcache structures obtained from system.
# TYPE go_memstats_mcache_sys_bytes gauge
go_memstats_mcache_sys_bytes 32768
# HELP go_memstats_mspan_inuse_bytes Number of bytes in use by mspan structures.
# TYPE go_memstats_mspan_inuse_bytes gauge
go_memstats_mspan_inuse_bytes 102272
# HELP go_memstats_mspan_sys_bytes Number of bytes used for mspan structures obtained from system.
# TYPE go_memstats_mspan_sys_bytes gauge
go_memstats_mspan_sys_bytes 131072
# HELP go_memstats_next_gc_bytes Number of heap bytes when next garbage collection will take place.
# TYPE go_memstats_next_gc_bytes gauge
go_memstats_next_gc_bytes 8.02544e+06
# HELP go_memstats_other_sys_bytes Number of bytes used for other system allocations.
# TYPE go_memstats_other_sys_bytes gauge
go_memstats_other_sys_bytes 2.490281e+06
# HELP go_memstats_stack_inuse_bytes Number of bytes in use by the stack allocator.
# TYPE go_memstats_stack_inuse_bytes gauge
go_memstats_stack_inuse_bytes 1.343488e+06
# HELP go_memstats_stack_sys_bytes Number of bytes obtained from system for stack allocator.
# TYPE go_memstats_stack_sys_bytes gauge
go_memstats_stack_sys_bytes 1.343488e+06
# HELP go_memstats_sys_bytes Number of bytes obtained from system.
# TYPE go_memstats_sys_bytes gauge
go_memstats_sys_bytes 7.3597176e+07
# HELP go_threads Number of OS threads created.
# TYPE go_threads gauge
go_threads 28
Metrics in active job state for main branch 
# HELP gitlab_runner_api_request_statuses_total The total number of api requests, partitioned by runner, endpoint and status.
# TYPE gitlab_runner_api_request_statuses_total counter
gitlab_runner_api_request_statuses_total{endpoint="request_job",runner="1-DdQors",status="204"} 1
gitlab_runner_api_request_statuses_total{endpoint="request_job",runner="8S27vskH",status="204"} 1
gitlab_runner_api_request_statuses_total{endpoint="request_job",runner="USPj1gp5",status="204"} 1
gitlab_runner_api_request_statuses_total{endpoint="request_job",runner="cpwhDr7z",status="201"} 1
gitlab_runner_api_request_statuses_total{endpoint="request_job",runner="cpwhDr7z",status="204"} 1
gitlab_runner_api_request_statuses_total{endpoint="request_job",runner="my7zCavq",status="204"} 1
gitlab_runner_api_request_statuses_total{endpoint="request_job",runner="zqMtkFxL",status="204"} 1
# HELP gitlab_runner_autoscaling_machine_creation_duration_seconds Histogram of machine creation time.
# TYPE gitlab_runner_autoscaling_machine_creation_duration_seconds histogram
gitlab_runner_autoscaling_machine_creation_duration_seconds_bucket{executor="docker+machine",le="30"} 0
gitlab_runner_autoscaling_machine_creation_duration_seconds_bucket{executor="docker+machine",le="37.5"} 0
gitlab_runner_autoscaling_machine_creation_duration_seconds_bucket{executor="docker+machine",le="46.875"} 0
gitlab_runner_autoscaling_machine_creation_duration_seconds_bucket{executor="docker+machine",le="58.59375"} 0
gitlab_runner_autoscaling_machine_creation_duration_seconds_bucket{executor="docker+machine",le="73.2421875"} 0
gitlab_runner_autoscaling_machine_creation_duration_seconds_bucket{executor="docker+machine",le="91.552734375"} 0
gitlab_runner_autoscaling_machine_creation_duration_seconds_bucket{executor="docker+machine",le="114.44091796875"} 0
gitlab_runner_autoscaling_machine_creation_duration_seconds_bucket{executor="docker+machine",le="143.0511474609375"} 0
gitlab_runner_autoscaling_machine_creation_duration_seconds_bucket{executor="docker+machine",le="178.81393432617188"} 0
gitlab_runner_autoscaling_machine_creation_duration_seconds_bucket{executor="docker+machine",le="223.51741790771484"} 0
gitlab_runner_autoscaling_machine_creation_duration_seconds_bucket{executor="docker+machine",le="+Inf"} 0
gitlab_runner_autoscaling_machine_creation_duration_seconds_sum{executor="docker+machine"} 0
gitlab_runner_autoscaling_machine_creation_duration_seconds_count{executor="docker+machine"} 0
gitlab_runner_autoscaling_machine_creation_duration_seconds_bucket{executor="docker-ssh+machine",le="30"} 0
gitlab_runner_autoscaling_machine_creation_duration_seconds_bucket{executor="docker-ssh+machine",le="37.5"} 0
gitlab_runner_autoscaling_machine_creation_duration_seconds_bucket{executor="docker-ssh+machine",le="46.875"} 0
gitlab_runner_autoscaling_machine_creation_duration_seconds_bucket{executor="docker-ssh+machine",le="58.59375"} 0
gitlab_runner_autoscaling_machine_creation_duration_seconds_bucket{executor="docker-ssh+machine",le="73.2421875"} 0
gitlab_runner_autoscaling_machine_creation_duration_seconds_bucket{executor="docker-ssh+machine",le="91.552734375"} 0
gitlab_runner_autoscaling_machine_creation_duration_seconds_bucket{executor="docker-ssh+machine",le="114.44091796875"} 0
gitlab_runner_autoscaling_machine_creation_duration_seconds_bucket{executor="docker-ssh+machine",le="143.0511474609375"} 0
gitlab_runner_autoscaling_machine_creation_duration_seconds_bucket{executor="docker-ssh+machine",le="178.81393432617188"} 0
gitlab_runner_autoscaling_machine_creation_duration_seconds_bucket{executor="docker-ssh+machine",le="223.51741790771484"} 0
gitlab_runner_autoscaling_machine_creation_duration_seconds_bucket{executor="docker-ssh+machine",le="+Inf"} 0
gitlab_runner_autoscaling_machine_creation_duration_seconds_sum{executor="docker-ssh+machine"} 0
gitlab_runner_autoscaling_machine_creation_duration_seconds_count{executor="docker-ssh+machine"} 0
# HELP gitlab_runner_autoscaling_machine_removal_duration_seconds Histogram of machine removal time.
# TYPE gitlab_runner_autoscaling_machine_removal_duration_seconds histogram
gitlab_runner_autoscaling_machine_removal_duration_seconds_bucket{executor="docker+machine",le="1"} 0
gitlab_runner_autoscaling_machine_removal_duration_seconds_bucket{executor="docker+machine",le="3"} 0
gitlab_runner_autoscaling_machine_removal_duration_seconds_bucket{executor="docker+machine",le="5"} 0
gitlab_runner_autoscaling_machine_removal_duration_seconds_bucket{executor="docker+machine",le="10"} 0
gitlab_runner_autoscaling_machine_removal_duration_seconds_bucket{executor="docker+machine",le="30"} 0
gitlab_runner_autoscaling_machine_removal_duration_seconds_bucket{executor="docker+machine",le="50"} 0
gitlab_runner_autoscaling_machine_removal_duration_seconds_bucket{executor="docker+machine",le="60"} 0
gitlab_runner_autoscaling_machine_removal_duration_seconds_bucket{executor="docker+machine",le="80"} 0
gitlab_runner_autoscaling_machine_removal_duration_seconds_bucket{executor="docker+machine",le="90"} 0
gitlab_runner_autoscaling_machine_removal_duration_seconds_bucket{executor="docker+machine",le="120"} 0
gitlab_runner_autoscaling_machine_removal_duration_seconds_bucket{executor="docker+machine",le="+Inf"} 0
gitlab_runner_autoscaling_machine_removal_duration_seconds_sum{executor="docker+machine"} 0
gitlab_runner_autoscaling_machine_removal_duration_seconds_count{executor="docker+machine"} 0
gitlab_runner_autoscaling_machine_removal_duration_seconds_bucket{executor="docker-ssh+machine",le="1"} 0
gitlab_runner_autoscaling_machine_removal_duration_seconds_bucket{executor="docker-ssh+machine",le="3"} 0
gitlab_runner_autoscaling_machine_removal_duration_seconds_bucket{executor="docker-ssh+machine",le="5"} 0
gitlab_runner_autoscaling_machine_removal_duration_seconds_bucket{executor="docker-ssh+machine",le="10"} 0
gitlab_runner_autoscaling_machine_removal_duration_seconds_bucket{executor="docker-ssh+machine",le="30"} 0
gitlab_runner_autoscaling_machine_removal_duration_seconds_bucket{executor="docker-ssh+machine",le="50"} 0
gitlab_runner_autoscaling_machine_removal_duration_seconds_bucket{executor="docker-ssh+machine",le="60"} 0
gitlab_runner_autoscaling_machine_removal_duration_seconds_bucket{executor="docker-ssh+machine",le="80"} 0
gitlab_runner_autoscaling_machine_removal_duration_seconds_bucket{executor="docker-ssh+machine",le="90"} 0
gitlab_runner_autoscaling_machine_removal_duration_seconds_bucket{executor="docker-ssh+machine",le="120"} 0
gitlab_runner_autoscaling_machine_removal_duration_seconds_bucket{executor="docker-ssh+machine",le="+Inf"} 0
gitlab_runner_autoscaling_machine_removal_duration_seconds_sum{executor="docker-ssh+machine"} 0
gitlab_runner_autoscaling_machine_removal_duration_seconds_count{executor="docker-ssh+machine"} 0
# HELP gitlab_runner_autoscaling_machine_states The current number of machines per state in this provider.
# TYPE gitlab_runner_autoscaling_machine_states gauge
gitlab_runner_autoscaling_machine_states{executor="docker+machine",state="acquired"} 0
gitlab_runner_autoscaling_machine_states{executor="docker+machine",state="creating"} 0
gitlab_runner_autoscaling_machine_states{executor="docker+machine",state="idle"} 0
gitlab_runner_autoscaling_machine_states{executor="docker+machine",state="removing"} 0
gitlab_runner_autoscaling_machine_states{executor="docker+machine",state="stuck-on-removing"} 0
gitlab_runner_autoscaling_machine_states{executor="docker+machine",state="used"} 0
gitlab_runner_autoscaling_machine_states{executor="docker-ssh+machine",state="acquired"} 0
gitlab_runner_autoscaling_machine_states{executor="docker-ssh+machine",state="creating"} 0
gitlab_runner_autoscaling_machine_states{executor="docker-ssh+machine",state="idle"} 0
gitlab_runner_autoscaling_machine_states{executor="docker-ssh+machine",state="removing"} 0
gitlab_runner_autoscaling_machine_states{executor="docker-ssh+machine",state="stuck-on-removing"} 0
gitlab_runner_autoscaling_machine_states{executor="docker-ssh+machine",state="used"} 0
# HELP gitlab_runner_autoscaling_machine_stopping_duration_seconds Histogram of machine stopping time.
# TYPE gitlab_runner_autoscaling_machine_stopping_duration_seconds histogram
gitlab_runner_autoscaling_machine_stopping_duration_seconds_bucket{executor="docker+machine",le="1"} 0
gitlab_runner_autoscaling_machine_stopping_duration_seconds_bucket{executor="docker+machine",le="3"} 0
gitlab_runner_autoscaling_machine_stopping_duration_seconds_bucket{executor="docker+machine",le="5"} 0
gitlab_runner_autoscaling_machine_stopping_duration_seconds_bucket{executor="docker+machine",le="10"} 0
gitlab_runner_autoscaling_machine_stopping_duration_seconds_bucket{executor="docker+machine",le="30"} 0
gitlab_runner_autoscaling_machine_stopping_duration_seconds_bucket{executor="docker+machine",le="50"} 0
gitlab_runner_autoscaling_machine_stopping_duration_seconds_bucket{executor="docker+machine",le="60"} 0
gitlab_runner_autoscaling_machine_stopping_duration_seconds_bucket{executor="docker+machine",le="80"} 0
gitlab_runner_autoscaling_machine_stopping_duration_seconds_bucket{executor="docker+machine",le="90"} 0
gitlab_runner_autoscaling_machine_stopping_duration_seconds_bucket{executor="docker+machine",le="120"} 0
gitlab_runner_autoscaling_machine_stopping_duration_seconds_bucket{executor="docker+machine",le="+Inf"} 0
gitlab_runner_autoscaling_machine_stopping_duration_seconds_sum{executor="docker+machine"} 0
gitlab_runner_autoscaling_machine_stopping_duration_seconds_count{executor="docker+machine"} 0
gitlab_runner_autoscaling_machine_stopping_duration_seconds_bucket{executor="docker-ssh+machine",le="1"} 0
gitlab_runner_autoscaling_machine_stopping_duration_seconds_bucket{executor="docker-ssh+machine",le="3"} 0
gitlab_runner_autoscaling_machine_stopping_duration_seconds_bucket{executor="docker-ssh+machine",le="5"} 0
gitlab_runner_autoscaling_machine_stopping_duration_seconds_bucket{executor="docker-ssh+machine",le="10"} 0
gitlab_runner_autoscaling_machine_stopping_duration_seconds_bucket{executor="docker-ssh+machine",le="30"} 0
gitlab_runner_autoscaling_machine_stopping_duration_seconds_bucket{executor="docker-ssh+machine",le="50"} 0
gitlab_runner_autoscaling_machine_stopping_duration_seconds_bucket{executor="docker-ssh+machine",le="60"} 0
gitlab_runner_autoscaling_machine_stopping_duration_seconds_bucket{executor="docker-ssh+machine",le="80"} 0
gitlab_runner_autoscaling_machine_stopping_duration_seconds_bucket{executor="docker-ssh+machine",le="90"} 0
gitlab_runner_autoscaling_machine_stopping_duration_seconds_bucket{executor="docker-ssh+machine",le="120"} 0
gitlab_runner_autoscaling_machine_stopping_duration_seconds_bucket{executor="docker-ssh+machine",le="+Inf"} 0
gitlab_runner_autoscaling_machine_stopping_duration_seconds_sum{executor="docker-ssh+machine"} 0
gitlab_runner_autoscaling_machine_stopping_duration_seconds_count{executor="docker-ssh+machine"} 0
# HELP gitlab_runner_concurrent The current value of concurrent setting
# TYPE gitlab_runner_concurrent gauge
gitlab_runner_concurrent 4
# HELP gitlab_runner_errors_total The number of caught errors.
# TYPE gitlab_runner_errors_total counter
gitlab_runner_errors_total{level="error"} 2
gitlab_runner_errors_total{level="fatal"} 0
gitlab_runner_errors_total{level="panic"} 0
gitlab_runner_errors_total{level="warning"} 7
# HELP gitlab_runner_jobs The current number of running builds.
# TYPE gitlab_runner_jobs gauge
gitlab_runner_jobs{executor_stage="cleanup",runner="cpwhDr7z",stage="",state="pending"} 1
# HELP gitlab_runner_jobs_total Total number of handled jobs
# TYPE gitlab_runner_jobs_total counter
gitlab_runner_jobs_total{runner="cpwhDr7z"} 1
# HELP gitlab_runner_limit The current value of concurrent setting
# TYPE gitlab_runner_limit gauge
gitlab_runner_limit{runner="1-DdQors"} 0
gitlab_runner_limit{runner="8S27vskH"} 0
gitlab_runner_limit{runner="USPj1gp5"} 0
gitlab_runner_limit{runner="cpwhDr7z"} 0
gitlab_runner_limit{runner="my7zCavq"} 0
gitlab_runner_limit{runner="zqMtkFxL"} 0
# HELP gitlab_runner_request_concurrency The current number of concurrent requests for a new job
# TYPE gitlab_runner_request_concurrency gauge
gitlab_runner_request_concurrency{runner="1-DdQors"} 0
gitlab_runner_request_concurrency{runner="8S27vskH"} 0
gitlab_runner_request_concurrency{runner="USPj1gp5"} 1
gitlab_runner_request_concurrency{runner="cpwhDr7z"} 1
gitlab_runner_request_concurrency{runner="my7zCavq"} 0
gitlab_runner_request_concurrency{runner="zqMtkFxL"} 1
# HELP gitlab_runner_request_concurrency_exceeded_total Counter tracking exceeding of request concurrency
# TYPE gitlab_runner_request_concurrency_exceeded_total counter
gitlab_runner_request_concurrency_exceeded_total{runner="1-DdQors"} 0
gitlab_runner_request_concurrency_exceeded_total{runner="8S27vskH"} 0
gitlab_runner_request_concurrency_exceeded_total{runner="USPj1gp5"} 0
gitlab_runner_request_concurrency_exceeded_total{runner="cpwhDr7z"} 0
gitlab_runner_request_concurrency_exceeded_total{runner="my7zCavq"} 0
gitlab_runner_request_concurrency_exceeded_total{runner="zqMtkFxL"} 0
# HELP gitlab_runner_version_info A metric with a constant '1' value labeled by different build stats fields.
# TYPE gitlab_runner_version_info gauge
gitlab_runner_version_info{architecture="amd64",branch="HEAD",built_at="unknown",go_version="go1.13.8",name="gitlab-runner",os="darwin",revision="HEAD",version="development version"} 1
# HELP go_gc_duration_seconds A summary of the GC invocation durations.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 2.6997e-05
go_gc_duration_seconds{quantile="0.25"} 3.9972e-05
go_gc_duration_seconds{quantile="0.5"} 6.9385e-05
go_gc_duration_seconds{quantile="0.75"} 9.2344e-05
go_gc_duration_seconds{quantile="1"} 0.000167532
go_gc_duration_seconds_sum 0.000774369
go_gc_duration_seconds_count 11
# HELP go_goroutines Number of goroutines that currently exist.
# TYPE go_goroutines gauge
go_goroutines 42
# HELP go_info Information about the Go environment.
# TYPE go_info gauge
go_info{version="go1.13.8"} 1
# HELP go_memstats_alloc_bytes Number of bytes allocated and still in use.
# TYPE go_memstats_alloc_bytes gauge
go_memstats_alloc_bytes 4.685912e+06
# HELP go_memstats_alloc_bytes_total Total number of bytes allocated, even if freed.
# TYPE go_memstats_alloc_bytes_total counter
go_memstats_alloc_bytes_total 1.3084656e+07
# HELP go_memstats_buck_hash_sys_bytes Number of bytes used by the profiling bucket hash table.
# TYPE go_memstats_buck_hash_sys_bytes gauge
go_memstats_buck_hash_sys_bytes 1.448271e+06
# HELP go_memstats_frees_total Total number of frees.
# TYPE go_memstats_frees_total counter
go_memstats_frees_total 114499
# HELP go_memstats_gc_cpu_fraction The fraction of this program's available CPU time used by the GC since the program started.
# TYPE go_memstats_gc_cpu_fraction gauge
go_memstats_gc_cpu_fraction 0.0007963798513790878
# HELP go_memstats_gc_sys_bytes Number of bytes used for garbage collection system metadata.
# TYPE go_memstats_gc_sys_bytes gauge
go_memstats_gc_sys_bytes 2.38592e+06
# HELP go_memstats_heap_alloc_bytes Number of heap bytes allocated and still in use.
# TYPE go_memstats_heap_alloc_bytes gauge
go_memstats_heap_alloc_bytes 4.685912e+06
# HELP go_memstats_heap_idle_bytes Number of heap bytes waiting to be used.
# TYPE go_memstats_heap_idle_bytes gauge
go_memstats_heap_idle_bytes 5.8466304e+07
# HELP go_memstats_heap_inuse_bytes Number of heap bytes that are in use.
# TYPE go_memstats_heap_inuse_bytes gauge
go_memstats_heap_inuse_bytes 7.33184e+06
# HELP go_memstats_heap_objects Number of allocated objects.
# TYPE go_memstats_heap_objects gauge
go_memstats_heap_objects 32171
# HELP go_memstats_heap_released_bytes Number of heap bytes released to OS.
# TYPE go_memstats_heap_released_bytes gauge
go_memstats_heap_released_bytes 5.820416e+07
# HELP go_memstats_heap_sys_bytes Number of heap bytes obtained from system.
# TYPE go_memstats_heap_sys_bytes gauge
go_memstats_heap_sys_bytes 6.5798144e+07
# HELP go_memstats_last_gc_time_seconds Number of seconds since 1970 of last garbage collection.
# TYPE go_memstats_last_gc_time_seconds gauge
go_memstats_last_gc_time_seconds 1.636731785055924e+09
# HELP go_memstats_lookups_total Total number of pointer lookups.
# TYPE go_memstats_lookups_total counter
go_memstats_lookups_total 0
# HELP go_memstats_mallocs_total Total number of mallocs.
# TYPE go_memstats_mallocs_total counter
go_memstats_mallocs_total 146670
# HELP go_memstats_mcache_inuse_bytes Number of bytes in use by mcache structures.
# TYPE go_memstats_mcache_inuse_bytes gauge
go_memstats_mcache_inuse_bytes 27776
# HELP go_memstats_mcache_sys_bytes Number of bytes used for mcache structures obtained from system.
# TYPE go_memstats_mcache_sys_bytes gauge
go_memstats_mcache_sys_bytes 32768
# HELP go_memstats_mspan_inuse_bytes Number of bytes in use by mspan structures.
# TYPE go_memstats_mspan_inuse_bytes gauge
go_memstats_mspan_inuse_bytes 99552
# HELP go_memstats_mspan_sys_bytes Number of bytes used for mspan structures obtained from system.
# TYPE go_memstats_mspan_sys_bytes gauge
go_memstats_mspan_sys_bytes 114688
# HELP go_memstats_next_gc_bytes Number of heap bytes when next garbage collection will take place.
# TYPE go_memstats_next_gc_bytes gauge
go_memstats_next_gc_bytes 8.512816e+06
# HELP go_memstats_other_sys_bytes Number of bytes used for other system allocations.
# TYPE go_memstats_other_sys_bytes gauge
go_memstats_other_sys_bytes 1.720233e+06
# HELP go_memstats_stack_inuse_bytes Number of bytes in use by the stack allocator.
# TYPE go_memstats_stack_inuse_bytes gauge
go_memstats_stack_inuse_bytes 1.31072e+06
# HELP go_memstats_stack_sys_bytes Number of bytes obtained from system for stack allocator.
# TYPE go_memstats_stack_sys_bytes gauge
go_memstats_stack_sys_bytes 1.31072e+06
# HELP go_memstats_sys_bytes Number of bytes obtained from system.
# TYPE go_memstats_sys_bytes gauge
go_memstats_sys_bytes 7.2810744e+07
# HELP go_threads Number of OS threads created.
# TYPE go_threads gauge
go_threads 27
Metrics in idle state for this branch 
# HELP gitlab_runner_api_request_statuses_total The total number of api requests, partitioned by runner, endpoint and status.
# TYPE gitlab_runner_api_request_statuses_total counter
gitlab_runner_api_request_statuses_total{endpoint="request_job",runner="1-DdQors",status="204"} 1
gitlab_runner_api_request_statuses_total{endpoint="request_job",runner="8S27vskH",status="204"} 1
gitlab_runner_api_request_statuses_total{endpoint="request_job",runner="USPj1gp5",status="204"} 1
gitlab_runner_api_request_statuses_total{endpoint="request_job",runner="cpwhDr7z",status="204"} 1
gitlab_runner_api_request_statuses_total{endpoint="request_job",runner="my7zCavq",status="204"} 1
gitlab_runner_api_request_statuses_total{endpoint="request_job",runner="zqMtkFxL",status="204"} 1
# HELP gitlab_runner_autoscaling_machine_creation_duration_seconds Histogram of machine creation time.
# TYPE gitlab_runner_autoscaling_machine_creation_duration_seconds histogram
gitlab_runner_autoscaling_machine_creation_duration_seconds_bucket{executor="docker+machine",le="30"} 0
gitlab_runner_autoscaling_machine_creation_duration_seconds_bucket{executor="docker+machine",le="37.5"} 0
gitlab_runner_autoscaling_machine_creation_duration_seconds_bucket{executor="docker+machine",le="46.875"} 0
gitlab_runner_autoscaling_machine_creation_duration_seconds_bucket{executor="docker+machine",le="58.59375"} 0
gitlab_runner_autoscaling_machine_creation_duration_seconds_bucket{executor="docker+machine",le="73.2421875"} 0
gitlab_runner_autoscaling_machine_creation_duration_seconds_bucket{executor="docker+machine",le="91.552734375"} 0
gitlab_runner_autoscaling_machine_creation_duration_seconds_bucket{executor="docker+machine",le="114.44091796875"} 0
gitlab_runner_autoscaling_machine_creation_duration_seconds_bucket{executor="docker+machine",le="143.0511474609375"} 0
gitlab_runner_autoscaling_machine_creation_duration_seconds_bucket{executor="docker+machine",le="178.81393432617188"} 0
gitlab_runner_autoscaling_machine_creation_duration_seconds_bucket{executor="docker+machine",le="223.51741790771484"} 0
gitlab_runner_autoscaling_machine_creation_duration_seconds_bucket{executor="docker+machine",le="+Inf"} 0
gitlab_runner_autoscaling_machine_creation_duration_seconds_sum{executor="docker+machine"} 0
gitlab_runner_autoscaling_machine_creation_duration_seconds_count{executor="docker+machine"} 0
gitlab_runner_autoscaling_machine_creation_duration_seconds_bucket{executor="docker-ssh+machine",le="30"} 0
gitlab_runner_autoscaling_machine_creation_duration_seconds_bucket{executor="docker-ssh+machine",le="37.5"} 0
gitlab_runner_autoscaling_machine_creation_duration_seconds_bucket{executor="docker-ssh+machine",le="46.875"} 0
gitlab_runner_autoscaling_machine_creation_duration_seconds_bucket{executor="docker-ssh+machine",le="58.59375"} 0
gitlab_runner_autoscaling_machine_creation_duration_seconds_bucket{executor="docker-ssh+machine",le="73.2421875"} 0
gitlab_runner_autoscaling_machine_creation_duration_seconds_bucket{executor="docker-ssh+machine",le="91.552734375"} 0
gitlab_runner_autoscaling_machine_creation_duration_seconds_bucket{executor="docker-ssh+machine",le="114.44091796875"} 0
gitlab_runner_autoscaling_machine_creation_duration_seconds_bucket{executor="docker-ssh+machine",le="143.0511474609375"} 0
gitlab_runner_autoscaling_machine_creation_duration_seconds_bucket{executor="docker-ssh+machine",le="178.81393432617188"} 0
gitlab_runner_autoscaling_machine_creation_duration_seconds_bucket{executor="docker-ssh+machine",le="223.51741790771484"} 0
gitlab_runner_autoscaling_machine_creation_duration_seconds_bucket{executor="docker-ssh+machine",le="+Inf"} 0
gitlab_runner_autoscaling_machine_creation_duration_seconds_sum{executor="docker-ssh+machine"} 0
gitlab_runner_autoscaling_machine_creation_duration_seconds_count{executor="docker-ssh+machine"} 0
# HELP gitlab_runner_autoscaling_machine_removal_duration_seconds Histogram of machine removal time.
# TYPE gitlab_runner_autoscaling_machine_removal_duration_seconds histogram
gitlab_runner_autoscaling_machine_removal_duration_seconds_bucket{executor="docker+machine",le="1"} 0
gitlab_runner_autoscaling_machine_removal_duration_seconds_bucket{executor="docker+machine",le="3"} 0
gitlab_runner_autoscaling_machine_removal_duration_seconds_bucket{executor="docker+machine",le="5"} 0
gitlab_runner_autoscaling_machine_removal_duration_seconds_bucket{executor="docker+machine",le="10"} 0
gitlab_runner_autoscaling_machine_removal_duration_seconds_bucket{executor="docker+machine",le="30"} 0
gitlab_runner_autoscaling_machine_removal_duration_seconds_bucket{executor="docker+machine",le="50"} 0
gitlab_runner_autoscaling_machine_removal_duration_seconds_bucket{executor="docker+machine",le="60"} 0
gitlab_runner_autoscaling_machine_removal_duration_seconds_bucket{executor="docker+machine",le="80"} 0
gitlab_runner_autoscaling_machine_removal_duration_seconds_bucket{executor="docker+machine",le="90"} 0
gitlab_runner_autoscaling_machine_removal_duration_seconds_bucket{executor="docker+machine",le="120"} 0
gitlab_runner_autoscaling_machine_removal_duration_seconds_bucket{executor="docker+machine",le="+Inf"} 0
gitlab_runner_autoscaling_machine_removal_duration_seconds_sum{executor="docker+machine"} 0
gitlab_runner_autoscaling_machine_removal_duration_seconds_count{executor="docker+machine"} 0
gitlab_runner_autoscaling_machine_removal_duration_seconds_bucket{executor="docker-ssh+machine",le="1"} 0
gitlab_runner_autoscaling_machine_removal_duration_seconds_bucket{executor="docker-ssh+machine",le="3"} 0
gitlab_runner_autoscaling_machine_removal_duration_seconds_bucket{executor="docker-ssh+machine",le="5"} 0
gitlab_runner_autoscaling_machine_removal_duration_seconds_bucket{executor="docker-ssh+machine",le="10"} 0
gitlab_runner_autoscaling_machine_removal_duration_seconds_bucket{executor="docker-ssh+machine",le="30"} 0
gitlab_runner_autoscaling_machine_removal_duration_seconds_bucket{executor="docker-ssh+machine",le="50"} 0
gitlab_runner_autoscaling_machine_removal_duration_seconds_bucket{executor="docker-ssh+machine",le="60"} 0
gitlab_runner_autoscaling_machine_removal_duration_seconds_bucket{executor="docker-ssh+machine",le="80"} 0
gitlab_runner_autoscaling_machine_removal_duration_seconds_bucket{executor="docker-ssh+machine",le="90"} 0
gitlab_runner_autoscaling_machine_removal_duration_seconds_bucket{executor="docker-ssh+machine",le="120"} 0
gitlab_runner_autoscaling_machine_removal_duration_seconds_bucket{executor="docker-ssh+machine",le="+Inf"} 0
gitlab_runner_autoscaling_machine_removal_duration_seconds_sum{executor="docker-ssh+machine"} 0
gitlab_runner_autoscaling_machine_removal_duration_seconds_count{executor="docker-ssh+machine"} 0
# HELP gitlab_runner_autoscaling_machine_states The current number of machines per state in this provider.
# TYPE gitlab_runner_autoscaling_machine_states gauge
gitlab_runner_autoscaling_machine_states{executor="docker+machine",state="acquired"} 0
gitlab_runner_autoscaling_machine_states{executor="docker+machine",state="creating"} 0
gitlab_runner_autoscaling_machine_states{executor="docker+machine",state="idle"} 0
gitlab_runner_autoscaling_machine_states{executor="docker+machine",state="removing"} 0
gitlab_runner_autoscaling_machine_states{executor="docker+machine",state="stuck-on-removing"} 0
gitlab_runner_autoscaling_machine_states{executor="docker+machine",state="used"} 0
gitlab_runner_autoscaling_machine_states{executor="docker-ssh+machine",state="acquired"} 0
gitlab_runner_autoscaling_machine_states{executor="docker-ssh+machine",state="creating"} 0
gitlab_runner_autoscaling_machine_states{executor="docker-ssh+machine",state="idle"} 0
gitlab_runner_autoscaling_machine_states{executor="docker-ssh+machine",state="removing"} 0
gitlab_runner_autoscaling_machine_states{executor="docker-ssh+machine",state="stuck-on-removing"} 0
gitlab_runner_autoscaling_machine_states{executor="docker-ssh+machine",state="used"} 0
# HELP gitlab_runner_autoscaling_machine_stopping_duration_seconds Histogram of machine stopping time.
# TYPE gitlab_runner_autoscaling_machine_stopping_duration_seconds histogram
gitlab_runner_autoscaling_machine_stopping_duration_seconds_bucket{executor="docker+machine",le="1"} 0
gitlab_runner_autoscaling_machine_stopping_duration_seconds_bucket{executor="docker+machine",le="3"} 0
gitlab_runner_autoscaling_machine_stopping_duration_seconds_bucket{executor="docker+machine",le="5"} 0
gitlab_runner_autoscaling_machine_stopping_duration_seconds_bucket{executor="docker+machine",le="10"} 0
gitlab_runner_autoscaling_machine_stopping_duration_seconds_bucket{executor="docker+machine",le="30"} 0
gitlab_runner_autoscaling_machine_stopping_duration_seconds_bucket{executor="docker+machine",le="50"} 0
gitlab_runner_autoscaling_machine_stopping_duration_seconds_bucket{executor="docker+machine",le="60"} 0
gitlab_runner_autoscaling_machine_stopping_duration_seconds_bucket{executor="docker+machine",le="80"} 0
gitlab_runner_autoscaling_machine_stopping_duration_seconds_bucket{executor="docker+machine",le="90"} 0
gitlab_runner_autoscaling_machine_stopping_duration_seconds_bucket{executor="docker+machine",le="120"} 0
gitlab_runner_autoscaling_machine_stopping_duration_seconds_bucket{executor="docker+machine",le="+Inf"} 0
gitlab_runner_autoscaling_machine_stopping_duration_seconds_sum{executor="docker+machine"} 0
gitlab_runner_autoscaling_machine_stopping_duration_seconds_count{executor="docker+machine"} 0
gitlab_runner_autoscaling_machine_stopping_duration_seconds_bucket{executor="docker-ssh+machine",le="1"} 0
gitlab_runner_autoscaling_machine_stopping_duration_seconds_bucket{executor="docker-ssh+machine",le="3"} 0
gitlab_runner_autoscaling_machine_stopping_duration_seconds_bucket{executor="docker-ssh+machine",le="5"} 0
gitlab_runner_autoscaling_machine_stopping_duration_seconds_bucket{executor="docker-ssh+machine",le="10"} 0
gitlab_runner_autoscaling_machine_stopping_duration_seconds_bucket{executor="docker-ssh+machine",le="30"} 0
gitlab_runner_autoscaling_machine_stopping_duration_seconds_bucket{executor="docker-ssh+machine",le="50"} 0
gitlab_runner_autoscaling_machine_stopping_duration_seconds_bucket{executor="docker-ssh+machine",le="60"} 0
gitlab_runner_autoscaling_machine_stopping_duration_seconds_bucket{executor="docker-ssh+machine",le="80"} 0
gitlab_runner_autoscaling_machine_stopping_duration_seconds_bucket{executor="docker-ssh+machine",le="90"} 0
gitlab_runner_autoscaling_machine_stopping_duration_seconds_bucket{executor="docker-ssh+machine",le="120"} 0
gitlab_runner_autoscaling_machine_stopping_duration_seconds_bucket{executor="docker-ssh+machine",le="+Inf"} 0
gitlab_runner_autoscaling_machine_stopping_duration_seconds_sum{executor="docker-ssh+machine"} 0
gitlab_runner_autoscaling_machine_stopping_duration_seconds_count{executor="docker-ssh+machine"} 0
# HELP gitlab_runner_concurrent The current value of concurrent setting
# TYPE gitlab_runner_concurrent gauge
gitlab_runner_concurrent 4
# HELP gitlab_runner_errors_total The number of caught errors.
# TYPE gitlab_runner_errors_total counter
gitlab_runner_errors_total{level="error"} 0
gitlab_runner_errors_total{level="fatal"} 0
gitlab_runner_errors_total{level="panic"} 0
gitlab_runner_errors_total{level="warning"} 5
# HELP gitlab_runner_jobs The current number of running builds.
# TYPE gitlab_runner_jobs gauge
gitlab_runner_jobs{executor_stage="idle",runner="1-DdQors",stage="idle",state="idle"} 0
gitlab_runner_jobs{executor_stage="idle",runner="8S27vskH",stage="idle",state="idle"} 0
gitlab_runner_jobs{executor_stage="idle",runner="USPj1gp5",stage="idle",state="idle"} 0
gitlab_runner_jobs{executor_stage="idle",runner="cpwhDr7z",stage="idle",state="idle"} 0
gitlab_runner_jobs{executor_stage="idle",runner="my7zCavq",stage="idle",state="idle"} 0
gitlab_runner_jobs{executor_stage="idle",runner="zqMtkFxL",stage="idle",state="idle"} 0
# HELP gitlab_runner_limit The current value of concurrent setting
# TYPE gitlab_runner_limit gauge
gitlab_runner_limit{runner="1-DdQors"} 0
gitlab_runner_limit{runner="8S27vskH"} 0
gitlab_runner_limit{runner="USPj1gp5"} 0
gitlab_runner_limit{runner="cpwhDr7z"} 0
gitlab_runner_limit{runner="my7zCavq"} 0
gitlab_runner_limit{runner="zqMtkFxL"} 0
# HELP gitlab_runner_request_concurrency The current number of concurrent requests for a new job
# TYPE gitlab_runner_request_concurrency gauge
gitlab_runner_request_concurrency{runner="1-DdQors"} 1
gitlab_runner_request_concurrency{runner="8S27vskH"} 0
gitlab_runner_request_concurrency{runner="USPj1gp5"} 1
gitlab_runner_request_concurrency{runner="cpwhDr7z"} 1
gitlab_runner_request_concurrency{runner="my7zCavq"} 0
gitlab_runner_request_concurrency{runner="zqMtkFxL"} 1
# HELP gitlab_runner_request_concurrency_exceeded_total Counter tracking exceeding of request concurrency
# TYPE gitlab_runner_request_concurrency_exceeded_total counter
gitlab_runner_request_concurrency_exceeded_total{runner="1-DdQors"} 0
gitlab_runner_request_concurrency_exceeded_total{runner="8S27vskH"} 0
gitlab_runner_request_concurrency_exceeded_total{runner="USPj1gp5"} 0
gitlab_runner_request_concurrency_exceeded_total{runner="cpwhDr7z"} 0
gitlab_runner_request_concurrency_exceeded_total{runner="my7zCavq"} 0
gitlab_runner_request_concurrency_exceeded_total{runner="zqMtkFxL"} 0
# HELP gitlab_runner_version_info A metric with a constant '1' value labeled by different build stats fields.
# TYPE gitlab_runner_version_info gauge
gitlab_runner_version_info{architecture="amd64",branch="HEAD",built_at="unknown",go_version="go1.13.8",name="gitlab-runner",os="darwin",revision="HEAD",version="development version"} 1
# HELP go_gc_duration_seconds A summary of the GC invocation durations.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 2.9233e-05
go_gc_duration_seconds{quantile="0.25"} 5.8414e-05
go_gc_duration_seconds{quantile="0.5"} 6.3362e-05
go_gc_duration_seconds{quantile="0.75"} 9.9844e-05
go_gc_duration_seconds{quantile="1"} 0.000181641
go_gc_duration_seconds_sum 0.000823276
go_gc_duration_seconds_count 10
# HELP go_goroutines Number of goroutines that currently exist.
# TYPE go_goroutines gauge
go_goroutines 40
# HELP go_info Information about the Go environment.
# TYPE go_info gauge
go_info{version="go1.13.8"} 1
# HELP go_memstats_alloc_bytes Number of bytes allocated and still in use.
# TYPE go_memstats_alloc_bytes gauge
go_memstats_alloc_bytes 4.134048e+06
# HELP go_memstats_alloc_bytes_total Total number of bytes allocated, even if freed.
# TYPE go_memstats_alloc_bytes_total counter
go_memstats_alloc_bytes_total 1.203364e+07
# HELP go_memstats_buck_hash_sys_bytes Number of bytes used by the profiling bucket hash table.
# TYPE go_memstats_buck_hash_sys_bytes gauge
go_memstats_buck_hash_sys_bytes 1.446391e+06
# HELP go_memstats_frees_total Total number of frees.
# TYPE go_memstats_frees_total counter
go_memstats_frees_total 110341
# HELP go_memstats_gc_cpu_fraction The fraction of this program's available CPU time used by the GC since the program started.
# TYPE go_memstats_gc_cpu_fraction gauge
go_memstats_gc_cpu_fraction 0.0004224316002202339
# HELP go_memstats_gc_sys_bytes Number of bytes used for garbage collection system metadata.
# TYPE go_memstats_gc_sys_bytes gauge
go_memstats_gc_sys_bytes 2.38592e+06
# HELP go_memstats_heap_alloc_bytes Number of heap bytes allocated and still in use.
# TYPE go_memstats_heap_alloc_bytes gauge
go_memstats_heap_alloc_bytes 4.134048e+06
# HELP go_memstats_heap_idle_bytes Number of heap bytes waiting to be used.
# TYPE go_memstats_heap_idle_bytes gauge
go_memstats_heap_idle_bytes 5.9047936e+07
# HELP go_memstats_heap_inuse_bytes Number of heap bytes that are in use.
# TYPE go_memstats_heap_inuse_bytes gauge
go_memstats_heap_inuse_bytes 6.750208e+06
# HELP go_memstats_heap_objects Number of allocated objects.
# TYPE go_memstats_heap_objects gauge
go_memstats_heap_objects 27058
# HELP go_memstats_heap_released_bytes Number of heap bytes released to OS.
# TYPE go_memstats_heap_released_bytes gauge
go_memstats_heap_released_bytes 5.8564608e+07
# HELP go_memstats_heap_sys_bytes Number of heap bytes obtained from system.
# TYPE go_memstats_heap_sys_bytes gauge
go_memstats_heap_sys_bytes 6.5798144e+07
# HELP go_memstats_last_gc_time_seconds Number of seconds since 1970 of last garbage collection.
# TYPE go_memstats_last_gc_time_seconds gauge
go_memstats_last_gc_time_seconds 1.636731507898457e+09
# HELP go_memstats_lookups_total Total number of pointer lookups.
# TYPE go_memstats_lookups_total counter
go_memstats_lookups_total 0
# HELP go_memstats_mallocs_total Total number of mallocs.
# TYPE go_memstats_mallocs_total counter
go_memstats_mallocs_total 137399
# HELP go_memstats_mcache_inuse_bytes Number of bytes in use by mcache structures.
# TYPE go_memstats_mcache_inuse_bytes gauge
go_memstats_mcache_inuse_bytes 27776
# HELP go_memstats_mcache_sys_bytes Number of bytes used for mcache structures obtained from system.
# TYPE go_memstats_mcache_sys_bytes gauge
go_memstats_mcache_sys_bytes 32768
# HELP go_memstats_mspan_inuse_bytes Number of bytes in use by mspan structures.
# TYPE go_memstats_mspan_inuse_bytes gauge
go_memstats_mspan_inuse_bytes 98464
# HELP go_memstats_mspan_sys_bytes Number of bytes used for mspan structures obtained from system.
# TYPE go_memstats_mspan_sys_bytes gauge
go_memstats_mspan_sys_bytes 114688
# HELP go_memstats_next_gc_bytes Number of heap bytes when next garbage collection will take place.
# TYPE go_memstats_next_gc_bytes gauge
go_memstats_next_gc_bytes 7.972048e+06
# HELP go_memstats_other_sys_bytes Number of bytes used for other system allocations.
# TYPE go_memstats_other_sys_bytes gauge
go_memstats_other_sys_bytes 1.984257e+06
# HELP go_memstats_stack_inuse_bytes Number of bytes in use by the stack allocator.
# TYPE go_memstats_stack_inuse_bytes gauge
go_memstats_stack_inuse_bytes 1.31072e+06
# HELP go_memstats_stack_sys_bytes Number of bytes obtained from system for stack allocator.
# TYPE go_memstats_stack_sys_bytes gauge
go_memstats_stack_sys_bytes 1.31072e+06
# HELP go_memstats_sys_bytes Number of bytes obtained from system.
# TYPE go_memstats_sys_bytes gauge
go_memstats_sys_bytes 7.3072888e+07
# HELP go_threads Number of OS threads created.
# TYPE go_threads gauge
go_threads 28
Metrics for active job state on this branch 
# HELP gitlab_runner_api_request_statuses_total The total number of api requests, partitioned by runner, endpoint and status.
# TYPE gitlab_runner_api_request_statuses_total counter
gitlab_runner_api_request_statuses_total{endpoint="patch_trace",runner="cpwhDr7z",status="202"} 2
gitlab_runner_api_request_statuses_total{endpoint="request_job",runner="1-DdQors",status="204"} 1
gitlab_runner_api_request_statuses_total{endpoint="request_job",runner="8S27vskH",status="204"} 1
gitlab_runner_api_request_statuses_total{endpoint="request_job",runner="USPj1gp5",status="204"} 1
gitlab_runner_api_request_statuses_total{endpoint="request_job",runner="cpwhDr7z",status="201"} 1
gitlab_runner_api_request_statuses_total{endpoint="request_job",runner="cpwhDr7z",status="204"} 1
gitlab_runner_api_request_statuses_total{endpoint="request_job",runner="my7zCavq",status="204"} 1
gitlab_runner_api_request_statuses_total{endpoint="request_job",runner="zqMtkFxL",status="204"} 1
# HELP gitlab_runner_autoscaling_machine_creation_duration_seconds Histogram of machine creation time.
# TYPE gitlab_runner_autoscaling_machine_creation_duration_seconds histogram
gitlab_runner_autoscaling_machine_creation_duration_seconds_bucket{executor="docker+machine",le="30"} 0
gitlab_runner_autoscaling_machine_creation_duration_seconds_bucket{executor="docker+machine",le="37.5"} 0
gitlab_runner_autoscaling_machine_creation_duration_seconds_bucket{executor="docker+machine",le="46.875"} 0
gitlab_runner_autoscaling_machine_creation_duration_seconds_bucket{executor="docker+machine",le="58.59375"} 0
gitlab_runner_autoscaling_machine_creation_duration_seconds_bucket{executor="docker+machine",le="73.2421875"} 0
gitlab_runner_autoscaling_machine_creation_duration_seconds_bucket{executor="docker+machine",le="91.552734375"} 0
gitlab_runner_autoscaling_machine_creation_duration_seconds_bucket{executor="docker+machine",le="114.44091796875"} 0
gitlab_runner_autoscaling_machine_creation_duration_seconds_bucket{executor="docker+machine",le="143.0511474609375"} 0
gitlab_runner_autoscaling_machine_creation_duration_seconds_bucket{executor="docker+machine",le="178.81393432617188"} 0
gitlab_runner_autoscaling_machine_creation_duration_seconds_bucket{executor="docker+machine",le="223.51741790771484"} 0
gitlab_runner_autoscaling_machine_creation_duration_seconds_bucket{executor="docker+machine",le="+Inf"} 0
gitlab_runner_autoscaling_machine_creation_duration_seconds_sum{executor="docker+machine"} 0
gitlab_runner_autoscaling_machine_creation_duration_seconds_count{executor="docker+machine"} 0
gitlab_runner_autoscaling_machine_creation_duration_seconds_bucket{executor="docker-ssh+machine",le="30"} 0
gitlab_runner_autoscaling_machine_creation_duration_seconds_bucket{executor="docker-ssh+machine",le="37.5"} 0
gitlab_runner_autoscaling_machine_creation_duration_seconds_bucket{executor="docker-ssh+machine",le="46.875"} 0
gitlab_runner_autoscaling_machine_creation_duration_seconds_bucket{executor="docker-ssh+machine",le="58.59375"} 0
gitlab_runner_autoscaling_machine_creation_duration_seconds_bucket{executor="docker-ssh+machine",le="73.2421875"} 0
gitlab_runner_autoscaling_machine_creation_duration_seconds_bucket{executor="docker-ssh+machine",le="91.552734375"} 0
gitlab_runner_autoscaling_machine_creation_duration_seconds_bucket{executor="docker-ssh+machine",le="114.44091796875"} 0
gitlab_runner_autoscaling_machine_creation_duration_seconds_bucket{executor="docker-ssh+machine",le="143.0511474609375"} 0
gitlab_runner_autoscaling_machine_creation_duration_seconds_bucket{executor="docker-ssh+machine",le="178.81393432617188"} 0
gitlab_runner_autoscaling_machine_creation_duration_seconds_bucket{executor="docker-ssh+machine",le="223.51741790771484"} 0
gitlab_runner_autoscaling_machine_creation_duration_seconds_bucket{executor="docker-ssh+machine",le="+Inf"} 0
gitlab_runner_autoscaling_machine_creation_duration_seconds_sum{executor="docker-ssh+machine"} 0
gitlab_runner_autoscaling_machine_creation_duration_seconds_count{executor="docker-ssh+machine"} 0
# HELP gitlab_runner_autoscaling_machine_removal_duration_seconds Histogram of machine removal time.
# TYPE gitlab_runner_autoscaling_machine_removal_duration_seconds histogram
gitlab_runner_autoscaling_machine_removal_duration_seconds_bucket{executor="docker+machine",le="1"} 0
gitlab_runner_autoscaling_machine_removal_duration_seconds_bucket{executor="docker+machine",le="3"} 0
gitlab_runner_autoscaling_machine_removal_duration_seconds_bucket{executor="docker+machine",le="5"} 0
gitlab_runner_autoscaling_machine_removal_duration_seconds_bucket{executor="docker+machine",le="10"} 0
gitlab_runner_autoscaling_machine_removal_duration_seconds_bucket{executor="docker+machine",le="30"} 0
gitlab_runner_autoscaling_machine_removal_duration_seconds_bucket{executor="docker+machine",le="50"} 0
gitlab_runner_autoscaling_machine_removal_duration_seconds_bucket{executor="docker+machine",le="60"} 0
gitlab_runner_autoscaling_machine_removal_duration_seconds_bucket{executor="docker+machine",le="80"} 0
gitlab_runner_autoscaling_machine_removal_duration_seconds_bucket{executor="docker+machine",le="90"} 0
gitlab_runner_autoscaling_machine_removal_duration_seconds_bucket{executor="docker+machine",le="120"} 0
gitlab_runner_autoscaling_machine_removal_duration_seconds_bucket{executor="docker+machine",le="+Inf"} 0
gitlab_runner_autoscaling_machine_removal_duration_seconds_sum{executor="docker+machine"} 0
gitlab_runner_autoscaling_machine_removal_duration_seconds_count{executor="docker+machine"} 0
gitlab_runner_autoscaling_machine_removal_duration_seconds_bucket{executor="docker-ssh+machine",le="1"} 0
gitlab_runner_autoscaling_machine_removal_duration_seconds_bucket{executor="docker-ssh+machine",le="3"} 0
gitlab_runner_autoscaling_machine_removal_duration_seconds_bucket{executor="docker-ssh+machine",le="5"} 0
gitlab_runner_autoscaling_machine_removal_duration_seconds_bucket{executor="docker-ssh+machine",le="10"} 0
gitlab_runner_autoscaling_machine_removal_duration_seconds_bucket{executor="docker-ssh+machine",le="30"} 0
gitlab_runner_autoscaling_machine_removal_duration_seconds_bucket{executor="docker-ssh+machine",le="50"} 0
gitlab_runner_autoscaling_machine_removal_duration_seconds_bucket{executor="docker-ssh+machine",le="60"} 0
gitlab_runner_autoscaling_machine_removal_duration_seconds_bucket{executor="docker-ssh+machine",le="80"} 0
gitlab_runner_autoscaling_machine_removal_duration_seconds_bucket{executor="docker-ssh+machine",le="90"} 0
gitlab_runner_autoscaling_machine_removal_duration_seconds_bucket{executor="docker-ssh+machine",le="120"} 0
gitlab_runner_autoscaling_machine_removal_duration_seconds_bucket{executor="docker-ssh+machine",le="+Inf"} 0
gitlab_runner_autoscaling_machine_removal_duration_seconds_sum{executor="docker-ssh+machine"} 0
gitlab_runner_autoscaling_machine_removal_duration_seconds_count{executor="docker-ssh+machine"} 0
# HELP gitlab_runner_autoscaling_machine_states The current number of machines per state in this provider.
# TYPE gitlab_runner_autoscaling_machine_states gauge
gitlab_runner_autoscaling_machine_states{executor="docker+machine",state="acquired"} 0
gitlab_runner_autoscaling_machine_states{executor="docker+machine",state="creating"} 0
gitlab_runner_autoscaling_machine_states{executor="docker+machine",state="idle"} 0
gitlab_runner_autoscaling_machine_states{executor="docker+machine",state="removing"} 0
gitlab_runner_autoscaling_machine_states{executor="docker+machine",state="stuck-on-removing"} 0
gitlab_runner_autoscaling_machine_states{executor="docker+machine",state="used"} 0
gitlab_runner_autoscaling_machine_states{executor="docker-ssh+machine",state="acquired"} 0
gitlab_runner_autoscaling_machine_states{executor="docker-ssh+machine",state="creating"} 0
gitlab_runner_autoscaling_machine_states{executor="docker-ssh+machine",state="idle"} 0
gitlab_runner_autoscaling_machine_states{executor="docker-ssh+machine",state="removing"} 0
gitlab_runner_autoscaling_machine_states{executor="docker-ssh+machine",state="stuck-on-removing"} 0
gitlab_runner_autoscaling_machine_states{executor="docker-ssh+machine",state="used"} 0
# HELP gitlab_runner_autoscaling_machine_stopping_duration_seconds Histogram of machine stopping time.
# TYPE gitlab_runner_autoscaling_machine_stopping_duration_seconds histogram
gitlab_runner_autoscaling_machine_stopping_duration_seconds_bucket{executor="docker+machine",le="1"} 0
gitlab_runner_autoscaling_machine_stopping_duration_seconds_bucket{executor="docker+machine",le="3"} 0
gitlab_runner_autoscaling_machine_stopping_duration_seconds_bucket{executor="docker+machine",le="5"} 0
gitlab_runner_autoscaling_machine_stopping_duration_seconds_bucket{executor="docker+machine",le="10"} 0
gitlab_runner_autoscaling_machine_stopping_duration_seconds_bucket{executor="docker+machine",le="30"} 0
gitlab_runner_autoscaling_machine_stopping_duration_seconds_bucket{executor="docker+machine",le="50"} 0
gitlab_runner_autoscaling_machine_stopping_duration_seconds_bucket{executor="docker+machine",le="60"} 0
gitlab_runner_autoscaling_machine_stopping_duration_seconds_bucket{executor="docker+machine",le="80"} 0
gitlab_runner_autoscaling_machine_stopping_duration_seconds_bucket{executor="docker+machine",le="90"} 0
gitlab_runner_autoscaling_machine_stopping_duration_seconds_bucket{executor="docker+machine",le="120"} 0
gitlab_runner_autoscaling_machine_stopping_duration_seconds_bucket{executor="docker+machine",le="+Inf"} 0
gitlab_runner_autoscaling_machine_stopping_duration_seconds_sum{executor="docker+machine"} 0
gitlab_runner_autoscaling_machine_stopping_duration_seconds_count{executor="docker+machine"} 0
gitlab_runner_autoscaling_machine_stopping_duration_seconds_bucket{executor="docker-ssh+machine",le="1"} 0
gitlab_runner_autoscaling_machine_stopping_duration_seconds_bucket{executor="docker-ssh+machine",le="3"} 0
gitlab_runner_autoscaling_machine_stopping_duration_seconds_bucket{executor="docker-ssh+machine",le="5"} 0
gitlab_runner_autoscaling_machine_stopping_duration_seconds_bucket{executor="docker-ssh+machine",le="10"} 0
gitlab_runner_autoscaling_machine_stopping_duration_seconds_bucket{executor="docker-ssh+machine",le="30"} 0
gitlab_runner_autoscaling_machine_stopping_duration_seconds_bucket{executor="docker-ssh+machine",le="50"} 0
gitlab_runner_autoscaling_machine_stopping_duration_seconds_bucket{executor="docker-ssh+machine",le="60"} 0
gitlab_runner_autoscaling_machine_stopping_duration_seconds_bucket{executor="docker-ssh+machine",le="80"} 0
gitlab_runner_autoscaling_machine_stopping_duration_seconds_bucket{executor="docker-ssh+machine",le="90"} 0
gitlab_runner_autoscaling_machine_stopping_duration_seconds_bucket{executor="docker-ssh+machine",le="120"} 0
gitlab_runner_autoscaling_machine_stopping_duration_seconds_bucket{executor="docker-ssh+machine",le="+Inf"} 0
gitlab_runner_autoscaling_machine_stopping_duration_seconds_sum{executor="docker-ssh+machine"} 0
gitlab_runner_autoscaling_machine_stopping_duration_seconds_count{executor="docker-ssh+machine"} 0
# HELP gitlab_runner_concurrent The current value of concurrent setting
# TYPE gitlab_runner_concurrent gauge
gitlab_runner_concurrent 4
# HELP gitlab_runner_errors_total The number of caught errors.
# TYPE gitlab_runner_errors_total counter
gitlab_runner_errors_total{level="error"} 3
gitlab_runner_errors_total{level="fatal"} 0
gitlab_runner_errors_total{level="panic"} 0
gitlab_runner_errors_total{level="warning"} 8
# HELP gitlab_runner_jobs The current number of running builds.
# TYPE gitlab_runner_jobs gauge
gitlab_runner_jobs{executor_stage="cleanup",runner="cpwhDr7z",stage="",state="pending"} 1
gitlab_runner_jobs{executor_stage="idle",runner="1-DdQors",stage="idle",state="idle"} 0
gitlab_runner_jobs{executor_stage="idle",runner="8S27vskH",stage="idle",state="idle"} 0
gitlab_runner_jobs{executor_stage="idle",runner="USPj1gp5",stage="idle",state="idle"} 0
gitlab_runner_jobs{executor_stage="idle",runner="cpwhDr7z",stage="idle",state="idle"} 0
gitlab_runner_jobs{executor_stage="idle",runner="my7zCavq",stage="idle",state="idle"} 0
gitlab_runner_jobs{executor_stage="idle",runner="zqMtkFxL",stage="idle",state="idle"} 0
# HELP gitlab_runner_jobs_total Total number of handled jobs
# TYPE gitlab_runner_jobs_total counter
gitlab_runner_jobs_total{runner="cpwhDr7z"} 1
# HELP gitlab_runner_limit The current value of concurrent setting
# TYPE gitlab_runner_limit gauge
gitlab_runner_limit{runner="1-DdQors"} 0
gitlab_runner_limit{runner="8S27vskH"} 0
gitlab_runner_limit{runner="USPj1gp5"} 0
gitlab_runner_limit{runner="cpwhDr7z"} 0
gitlab_runner_limit{runner="my7zCavq"} 0
gitlab_runner_limit{runner="zqMtkFxL"} 0
# HELP gitlab_runner_request_concurrency The current number of concurrent requests for a new job
# TYPE gitlab_runner_request_concurrency gauge
gitlab_runner_request_concurrency{runner="1-DdQors"} 0
gitlab_runner_request_concurrency{runner="8S27vskH"} 0
gitlab_runner_request_concurrency{runner="USPj1gp5"} 1
gitlab_runner_request_concurrency{runner="cpwhDr7z"} 1
gitlab_runner_request_concurrency{runner="my7zCavq"} 0
gitlab_runner_request_concurrency{runner="zqMtkFxL"} 1
# HELP gitlab_runner_request_concurrency_exceeded_total Counter tracking exceeding of request concurrency
# TYPE gitlab_runner_request_concurrency_exceeded_total counter
gitlab_runner_request_concurrency_exceeded_total{runner="1-DdQors"} 0
gitlab_runner_request_concurrency_exceeded_total{runner="8S27vskH"} 0
gitlab_runner_request_concurrency_exceeded_total{runner="USPj1gp5"} 0
gitlab_runner_request_concurrency_exceeded_total{runner="cpwhDr7z"} 0
gitlab_runner_request_concurrency_exceeded_total{runner="my7zCavq"} 0
gitlab_runner_request_concurrency_exceeded_total{runner="zqMtkFxL"} 0
# HELP gitlab_runner_version_info A metric with a constant '1' value labeled by different build stats fields.
# TYPE gitlab_runner_version_info gauge
gitlab_runner_version_info{architecture="amd64",branch="HEAD",built_at="unknown",go_version="go1.13.8",name="gitlab-runner",os="darwin",revision="HEAD",version="development version"} 1
# HELP go_gc_duration_seconds A summary of the GC invocation durations.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 2.8223e-05
go_gc_duration_seconds{quantile="0.25"} 3.6173e-05
go_gc_duration_seconds{quantile="0.5"} 6.6102e-05
go_gc_duration_seconds{quantile="0.75"} 9.364e-05
go_gc_duration_seconds{quantile="1"} 0.000106918
go_gc_duration_seconds_sum 0.000681132
go_gc_duration_seconds_count 10
# HELP go_goroutines Number of goroutines that currently exist.
# TYPE go_goroutines gauge
go_goroutines 42
# HELP go_info Information about the Go environment.
# TYPE go_info gauge
go_info{version="go1.13.8"} 1
# HELP go_memstats_alloc_bytes Number of bytes allocated and still in use.
# TYPE go_memstats_alloc_bytes gauge
go_memstats_alloc_bytes 4.738456e+06
# HELP go_memstats_alloc_bytes_total Total number of bytes allocated, even if freed.
# TYPE go_memstats_alloc_bytes_total counter
go_memstats_alloc_bytes_total 1.3081168e+07
# HELP go_memstats_buck_hash_sys_bytes Number of bytes used by the profiling bucket hash table.
# TYPE go_memstats_buck_hash_sys_bytes gauge
go_memstats_buck_hash_sys_bytes 1.445903e+06
# HELP go_memstats_frees_total Total number of frees.
# TYPE go_memstats_frees_total counter
go_memstats_frees_total 114438
# HELP go_memstats_gc_cpu_fraction The fraction of this program's available CPU time used by the GC since the program started.
# TYPE go_memstats_gc_cpu_fraction gauge
go_memstats_gc_cpu_fraction 0.0008226810322631783
# HELP go_memstats_gc_sys_bytes Number of bytes used for garbage collection system metadata.
# TYPE go_memstats_gc_sys_bytes gauge
go_memstats_gc_sys_bytes 2.38592e+06
# HELP go_memstats_heap_alloc_bytes Number of heap bytes allocated and still in use.
# TYPE go_memstats_heap_alloc_bytes gauge
go_memstats_heap_alloc_bytes 4.738456e+06
# HELP go_memstats_heap_idle_bytes Number of heap bytes waiting to be used.
# TYPE go_memstats_heap_idle_bytes gauge
go_memstats_heap_idle_bytes 5.881856e+07
# HELP go_memstats_heap_inuse_bytes Number of heap bytes that are in use.
# TYPE go_memstats_heap_inuse_bytes gauge
go_memstats_heap_inuse_bytes 6.946816e+06
# HELP go_memstats_heap_objects Number of allocated objects.
# TYPE go_memstats_heap_objects gauge
go_memstats_heap_objects 32826
# HELP go_memstats_heap_released_bytes Number of heap bytes released to OS.
# TYPE go_memstats_heap_released_bytes gauge
go_memstats_heap_released_bytes 5.8523648e+07
# HELP go_memstats_heap_sys_bytes Number of heap bytes obtained from system.
# TYPE go_memstats_heap_sys_bytes gauge
go_memstats_heap_sys_bytes 6.5765376e+07
# HELP go_memstats_last_gc_time_seconds Number of seconds since 1970 of last garbage collection.
# TYPE go_memstats_last_gc_time_seconds gauge
go_memstats_last_gc_time_seconds 1.636731622722086e+09
# HELP go_memstats_lookups_total Total number of pointer lookups.
# TYPE go_memstats_lookups_total counter
go_memstats_lookups_total 0
# HELP go_memstats_mallocs_total Total number of mallocs.
# TYPE go_memstats_mallocs_total counter
go_memstats_mallocs_total 147264
# HELP go_memstats_mcache_inuse_bytes Number of bytes in use by mcache structures.
# TYPE go_memstats_mcache_inuse_bytes gauge
go_memstats_mcache_inuse_bytes 27776
# HELP go_memstats_mcache_sys_bytes Number of bytes used for mcache structures obtained from system.
# TYPE go_memstats_mcache_sys_bytes gauge
go_memstats_mcache_sys_bytes 32768
# HELP go_memstats_mspan_inuse_bytes Number of bytes in use by mspan structures.
# TYPE go_memstats_mspan_inuse_bytes gauge
go_memstats_mspan_inuse_bytes 93160
# HELP go_memstats_mspan_sys_bytes Number of bytes used for mspan structures obtained from system.
# TYPE go_memstats_mspan_sys_bytes gauge
go_memstats_mspan_sys_bytes 98304
# HELP go_memstats_next_gc_bytes Number of heap bytes when next garbage collection will take place.
# TYPE go_memstats_next_gc_bytes gauge
go_memstats_next_gc_bytes 8.467584e+06
# HELP go_memstats_other_sys_bytes Number of bytes used for other system allocations.
# TYPE go_memstats_other_sys_bytes gauge
go_memstats_other_sys_bytes 1.738985e+06
# HELP go_memstats_stack_inuse_bytes Number of bytes in use by the stack allocator.
# TYPE go_memstats_stack_inuse_bytes gauge
go_memstats_stack_inuse_bytes 1.343488e+06
# HELP go_memstats_stack_sys_bytes Number of bytes obtained from system for stack allocator.
# TYPE go_memstats_stack_sys_bytes gauge
go_memstats_stack_sys_bytes 1.343488e+06
# HELP go_memstats_sys_bytes Number of bytes obtained from system.
# TYPE go_memstats_sys_bytes gauge
go_memstats_sys_bytes 7.2810744e+07
# HELP go_threads Number of OS threads created.
# TYPE go_threads gauge
go_threads 27

What are the relevant issue numbers?

Closes #25845 (closed)

Edited by Pedro Pombeiro (OOO from July 16th till Aug 7th)

Merge request reports