feat: add support for http.debug.tls for monitoring service
Related to Add support of TLS on `http.debug` (#729 - closed)
Depends on chore(configuration): add tls settings for http... (!1097 - merged)
Testing locally
A valid certificate and key are needed to configure TLS. An easy way to generate them is using the GDK:
# check the registry.host in your gdk.yml file
# the following command will recreate the certificates and the registry.host is used as the CN
rm -f registry_host.{key,crt} && make trust-docker-registryValid configurations:
- Both http and http.debug with tls enabled:
http:
addr: registry.test:5000
tls:
certificate: /Users/jaime/dev/gitlab/ee/registry_host.crt
key: /Users/jaime/dev/gitlab/ee/registry_host.key
debug:
addr: :5001
tls:
enabled: true
certificate: /Users/jaime/dev/gitlab/ee/registry_host.crt
key: /Users/jaime/dev/gitlab/ee/registry_host.key- Only http.tls with http.debug.tls enabled inherits config:
http:
addr: registry.test:5000
tls:
certificate: /Users/jaime/dev/gitlab/ee/registry_host.crt
key: /Users/jaime/dev/gitlab/ee/registry_host.key
debug:
addr: :5001
tls:
enabled: true- Only http.debug.tls enabled:
http:
addr: registry.test:5000
debug:
addr: :5001
tls:
enabled: true
certificate: /Users/jaime/dev/gitlab/ee/registry_host.crt
key: /Users/jaime/dev/gitlab/ee/registry_host.keyAll these configurations should allow to curl the metrics server with TLS:
curl examples
curl "http://registry.test:5001/metrics" ─╯
Client sent an HTTP request to an HTTPS server.
~/dev/gitlab/ee on main *2 ?2 ▓▒░············································································································································································░▒▓ at 18:00:40 ─╮
❯ curl "https://registry.test:5001/metrics" ─╯
curl: (60) SSL certificate problem: self signed certificate
More details here: https://curl.se/docs/sslcerts.html
curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
~/dev/gitlab/ee on main *2 ?2 ▓▒░············································································································································································░▒▓ at 18:00:43 ─╮
❯ curl "https://registry.test:5001/metrics" -k ─╯
# HELP gitlab_build_info Current build info for this GitLab Service
# TYPE gitlab_build_info gauge
gitlab_build_info{built="2022-08-01T08:00:09",package="github.com/docker/distribution",revision="72e82654161fdcaefcbcd60ec755df44707cac36",version="v3.54.0-gitlab-13-g72e82654"} 1
# HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 5.0086e-05
go_gc_duration_seconds{quantile="0.25"} 5.0086e-05
go_gc_duration_seconds{quantile="0.5"} 0.000175297
go_gc_duration_seconds{quantile="0.75"} 0.000187976
go_gc_duration_seconds{quantile="1"} 0.000187976
go_gc_duration_seconds_sum 0.000413359
go_gc_duration_seconds_count 3
# HELP go_goroutines Number of goroutines that currently exist.
# TYPE go_goroutines gauge
go_goroutines 16
# HELP go_info Information about the Go environment.
# TYPE go_info gauge
go_info{version="go1.17.8"} 1
# HELP go_memstats_alloc_bytes Number of bytes allocated and still in use.
# TYPE go_memstats_alloc_bytes gauge
go_memstats_alloc_bytes 6.387608e+06
# HELP go_memstats_alloc_bytes_total Total number of bytes allocated, even if freed.
# TYPE go_memstats_alloc_bytes_total counter
go_memstats_alloc_bytes_total 9.640912e+06
# HELP go_memstats_buck_hash_sys_bytes Number of bytes used by the profiling bucket hash table.
# TYPE go_memstats_buck_hash_sys_bytes gauge
go_memstats_buck_hash_sys_bytes 1.453922e+06
# HELP go_memstats_frees_total Total number of frees.
# TYPE go_memstats_frees_total counter
go_memstats_frees_total 26354
# HELP go_memstats_gc_sys_bytes Number of bytes used for garbage collection system metadata.
# TYPE go_memstats_gc_sys_bytes gauge
go_memstats_gc_sys_bytes 5.312288e+06
# HELP go_memstats_heap_alloc_bytes Number of heap bytes allocated and still in use.
# TYPE go_memstats_heap_alloc_bytes gauge
go_memstats_heap_alloc_bytes 6.387608e+06
# HELP go_memstats_heap_idle_bytes Number of heap bytes waiting to be used.
# TYPE go_memstats_heap_idle_bytes gauge
go_memstats_heap_idle_bytes 4.595712e+06
# HELP go_memstats_heap_inuse_bytes Number of heap bytes that are in use.
# TYPE go_memstats_heap_inuse_bytes gauge
go_memstats_heap_inuse_bytes 6.807552e+06
# HELP go_memstats_heap_objects Number of allocated objects.
# TYPE go_memstats_heap_objects gauge
go_memstats_heap_objects 53870
# HELP go_memstats_heap_released_bytes Number of heap bytes released to OS.
# TYPE go_memstats_heap_released_bytes gauge
go_memstats_heap_released_bytes 4.292608e+06
# HELP go_memstats_heap_sys_bytes Number of heap bytes obtained from system.
# TYPE go_memstats_heap_sys_bytes gauge
go_memstats_heap_sys_bytes 1.1403264e+07
# HELP go_memstats_last_gc_time_seconds Number of seconds since 1970 of last garbage collection.
# TYPE go_memstats_last_gc_time_seconds gauge
go_memstats_last_gc_time_seconds 1.659340812384724e+09
# HELP go_memstats_lookups_total Total number of pointer lookups.
# TYPE go_memstats_lookups_total counter
go_memstats_lookups_total 0
# HELP go_memstats_mallocs_total Total number of mallocs.
# TYPE go_memstats_mallocs_total counter
go_memstats_mallocs_total 80224
# HELP go_memstats_mcache_inuse_bytes Number of bytes in use by mcache structures.
# TYPE go_memstats_mcache_inuse_bytes gauge
go_memstats_mcache_inuse_bytes 19200
# HELP go_memstats_mcache_sys_bytes Number of bytes used for mcache structures obtained from system.
# TYPE go_memstats_mcache_sys_bytes gauge
go_memstats_mcache_sys_bytes 32768
# HELP go_memstats_mspan_inuse_bytes Number of bytes in use by mspan structures.
# TYPE go_memstats_mspan_inuse_bytes gauge
go_memstats_mspan_inuse_bytes 162520
# HELP go_memstats_mspan_sys_bytes Number of bytes used for mspan structures obtained from system.
# TYPE go_memstats_mspan_sys_bytes gauge
go_memstats_mspan_sys_bytes 163840
# HELP go_memstats_next_gc_bytes Number of heap bytes when next garbage collection will take place.
# TYPE go_memstats_next_gc_bytes gauge
go_memstats_next_gc_bytes 6.916096e+06
# HELP go_memstats_other_sys_bytes Number of bytes used for other system allocations.
# TYPE go_memstats_other_sys_bytes gauge
go_memstats_other_sys_bytes 2.03303e+06
# HELP go_memstats_stack_inuse_bytes Number of bytes in use by the stack allocator.
# TYPE go_memstats_stack_inuse_bytes gauge
go_memstats_stack_inuse_bytes 1.179648e+06
# HELP go_memstats_stack_sys_bytes Number of bytes obtained from system for stack allocator.
# TYPE go_memstats_stack_sys_bytes gauge
go_memstats_stack_sys_bytes 1.179648e+06
# HELP go_memstats_sys_bytes Number of bytes obtained from system.
# TYPE go_memstats_sys_bytes gauge
go_memstats_sys_bytes 2.157876e+07
# HELP go_sql_dbstats_connections_idle The number of idle connections.
# TYPE go_sql_dbstats_connections_idle gauge
go_sql_dbstats_connections_idle{db_name="registry_dev"} 1
# HELP go_sql_dbstats_connections_in_use The number of connections currently in use.
# TYPE go_sql_dbstats_connections_in_use gauge
go_sql_dbstats_connections_in_use{db_name="registry_dev"} 0
# HELP go_sql_dbstats_connections_max_idle_closed_count_total The total number of connections closed due to SetMaxIdleConns.
# TYPE go_sql_dbstats_connections_max_idle_closed_count_total counter
go_sql_dbstats_connections_max_idle_closed_count_total{db_name="registry_dev"} 0
# HELP go_sql_dbstats_connections_max_idle_time_closed_count_total The total number of connections closed due to SetConnMaxIdleTime.
# TYPE go_sql_dbstats_connections_max_idle_time_closed_count_total counter
go_sql_dbstats_connections_max_idle_time_closed_count_total{db_name="registry_dev"} 0
# HELP go_sql_dbstats_connections_max_lifetime_closed_count_total The total number of connections closed due to SetConnMaxLifetime.
# TYPE go_sql_dbstats_connections_max_lifetime_closed_count_total counter
go_sql_dbstats_connections_max_lifetime_closed_count_total{db_name="registry_dev"} 0
# HELP go_sql_dbstats_connections_max_open The limit of open connections to the database.
# TYPE go_sql_dbstats_connections_max_open gauge
go_sql_dbstats_connections_max_open{db_name="registry_dev"} 25
# HELP go_sql_dbstats_connections_open The number of established connections both in use and idle.
# TYPE go_sql_dbstats_connections_open gauge
go_sql_dbstats_connections_open{db_name="registry_dev"} 1
# HELP go_sql_dbstats_connections_wait_seconds_total The total time blocked waiting for a new connection.
# TYPE go_sql_dbstats_connections_wait_seconds_total counter
go_sql_dbstats_connections_wait_seconds_total{db_name="registry_dev"} 0
# HELP go_sql_dbstats_connections_waits_total The total number of connections waited for.
# TYPE go_sql_dbstats_connections_waits_total counter
go_sql_dbstats_connections_waits_total{db_name="registry_dev"} 0
# HELP go_threads Number of OS threads created.
# TYPE go_threads gauge
go_threads 18
# HELP promhttp_metric_handler_requests_in_flight Current number of scrapes being served.
# TYPE promhttp_metric_handler_requests_in_flight gauge
promhttp_metric_handler_requests_in_flight 1
# HELP promhttp_metric_handler_requests_total Total number of scrapes by HTTP status code.
# TYPE promhttp_metric_handler_requests_total counter
promhttp_metric_handler_requests_total{code="200"} 0
promhttp_metric_handler_requests_total{code="500"} 0
promhttp_metric_handler_requests_total{code="503"} 0
# HELP registry_http_import_worker_saturation A gauge of saturation of workers per instance.
# TYPE registry_http_import_worker_saturation gauge
registry_http_import_worker_saturation 0
# HELP registry_http_in_flight_requests A gauge of requests currently being served by the http server.
# TYPE registry_http_in_flight_requests gauge
registry_http_in_flight_requests 0
# HELP registry_migration_layer_counts A histogram of layer counts per (pre)imported manifest.
# TYPE registry_migration_layer_counts histogram
registry_migration_layer_counts_bucket{le="1"} 0
registry_migration_layer_counts_bucket{le="2"} 0
registry_migration_layer_counts_bucket{le="5"} 0
registry_migration_layer_counts_bucket{le="10"} 0
registry_migration_layer_counts_bucket{le="25"} 0
registry_migration_layer_counts_bucket{le="50"} 0
registry_migration_layer_counts_bucket{le="100"} 0
registry_migration_layer_counts_bucket{le="200"} 0
registry_migration_layer_counts_bucket{le="+Inf"} 0
registry_migration_layer_counts_sum 0
registry_migration_layer_counts_count 0
# HELP registry_notifications_pending_total The gauge of pending events in queue
# TYPE registry_notifications_pending_total gauge
registry_notifications_pending_total 0
# HELP registry_redis_pool_stats_hits The number of times a free connection was found in the pool.
# TYPE registry_redis_pool_stats_hits gauge
registry_redis_pool_stats_hits{instance="cache"} 0
# HELP registry_redis_pool_stats_idle_conns The number of idle connections in the pool.
# TYPE registry_redis_pool_stats_idle_conns gauge
registry_redis_pool_stats_idle_conns{instance="cache"} 1
# HELP registry_redis_pool_stats_misses The number of times a free connection was not found in the pool.
# TYPE registry_redis_pool_stats_misses gauge
registry_redis_pool_stats_misses{instance="cache"} 1
# HELP registry_redis_pool_stats_stale_conns The number of stale connections removed from the pool.
# TYPE registry_redis_pool_stats_stale_conns gauge
registry_redis_pool_stats_stale_conns{instance="cache"} 0
# HELP registry_redis_pool_stats_timeouts The number of times a wait timeout occurred.
# TYPE registry_redis_pool_stats_timeouts gauge
registry_redis_pool_stats_timeouts{instance="cache"} 0
# HELP registry_redis_pool_stats_total_conns The total number of connections in the pool.
# TYPE registry_redis_pool_stats_total_conns gauge
registry_redis_pool_stats_total_conns{instance="cache"} 1
# HELP registry_redis_single_commands Histogram of single Redis commands
# TYPE registry_redis_single_commands histogram
registry_redis_single_commands_bucket{command="ping",instance="cache",le="0.001"} 0
registry_redis_single_commands_bucket{command="ping",instance="cache",le="0.005"} 1
registry_redis_single_commands_bucket{command="ping",instance="cache",le="0.01"} 1
registry_redis_single_commands_bucket{command="ping",instance="cache",le="0.025"} 1
registry_redis_single_commands_bucket{command="ping",instance="cache",le="0.05"} 1
registry_redis_single_commands_bucket{command="ping",instance="cache",le="0.1"} 1
registry_redis_single_commands_bucket{command="ping",instance="cache",le="0.25"} 1
registry_redis_single_commands_bucket{command="ping",instance="cache",le="0.5"} 1
registry_redis_single_commands_bucket{command="ping",instance="cache",le="1"} 1
registry_redis_single_commands_bucket{command="ping",instance="cache",le="+Inf"} 1
registry_redis_single_commands_sum{command="ping",instance="cache"} 0.001315884
registry_redis_single_commands_count{command="ping",instance="cache"} 1
# HELP registry_storage_rate_limit_total A counter of requests to the storage driver that hit a rate limit.
# TYPE registry_storage_rate_limit_total counter
registry_storage_rate_limit_total 0Edited by Jaime Martinez