Skip to content

Add gitaly-blackbox prometheus exporter

Jacob Vosmaer requested to merge jv-gitaly-blackbox into master

Part of #1737 (closed)

This MR introduces a new Prometheus exporter: gitaly-blackbox. In this iteration it monitors exactly one thing, namely, how fast we can receive a Git clone from a HTTP URL.

This is more specific than doing an actual Git clone. We only count and consume the Git transport packets; we do not verify if we get a proper clone and we do not write anything to disk. This way of measuring should give a better indication of the maximum speed at which a Git server can serve up a Git HTTP clone, ignoring any possible bottlenecks in the client system.

Features include:

  • structured logging, with JSON support
  • configurable Prometheus listen address
  • configurable sleep time in the main probe loop

Not (yet) supported:

  • custom HTTP headers (e.g. to clone non-public repos)
  • concurrent probes
Log output

{"level":"info","msg":"starting probe","probe":"gitlab-test","time":"2020-02-25T16:15:25+01:00"}
{"level":"info","msg":"finished probe","probe":"gitlab-test","time":"2020-02-25T16:15:27+01:00"}
{"level":"info","msg":"starting probe","probe":"gitaly","time":"2020-02-25T16:15:27+01:00"}
{"level":"info","msg":"finished probe","probe":"gitaly","time":"2020-02-25T16:15:29+01:00"}
Metrics output

$ curl -s localhost:9999/metrics|grep ^gitaly
gitaly_blackbox_git_http_get_advertised_refs{probe="gitaly"} 3651
gitaly_blackbox_git_http_get_advertised_refs{probe="gitlab-test"} 162
gitaly_blackbox_git_http_get_first_packet{probe="gitaly"} 0.217950489
gitaly_blackbox_git_http_get_first_packet{probe="gitlab-test"} 0.182190911
gitaly_blackbox_git_http_get_total_time{probe="gitaly"} 0.240183232
gitaly_blackbox_git_http_get_total_time{probe="gitlab-test"} 0.182290058
gitaly_blackbox_git_http_post_first_pack_packet{probe="gitaly"} 0.592556218
gitaly_blackbox_git_http_post_first_pack_packet{probe="gitlab-test"} 0.288210009
gitaly_blackbox_git_http_post_first_progress_packet{probe="gitaly"} 0.570212166
gitaly_blackbox_git_http_post_first_progress_packet{probe="gitlab-test"} 0.28820953
gitaly_blackbox_git_http_post_pack_bytes{probe="gitaly"} 2.1555142e+07
gitaly_blackbox_git_http_post_pack_bytes{probe="gitlab-test"} 1.1208053e+07
gitaly_blackbox_git_http_post_total_time{probe="gitaly"} 1.89156588
gitaly_blackbox_git_http_post_total_time{probe="gitlab-test"} 1.085081146
gitaly_blackbox_git_http_wanted_refs{probe="gitaly"} 916
gitaly_blackbox_git_http_wanted_refs{probe="gitlab-test"} 104
Edited by GitLab Release Tools Bot

Merge request reports