gitlab_transaction_event_remote_mirrors metrics missing from gitlab-exporter
Summary
The metrics relative to remote mirroring are missing from the gitlab-exporter
Steps to reproduce
- Install gitlab via Helm chart https://gitlab.com/gitlab-org/charts/gitlab
- Enable gitlab-exporter
- Configure remote mirroring for some projects
- Configure external Prometheus scrape job to fetch metrics relative to the remote mirroring
What is the current bug behavior?
Some of the metrics from the gitlab-exporter are missing, the gitlab_transaction_event_remote_mirrors* ones
What is the expected correct behavior?
The exporter should expose all the metrics.
Relevant logs and/or screenshots
As far as we can see, the mirror process is working fine:
sidekiq logs:
2020-09-18T09:59:08.555Z 8 TID-gqwawidjc UpdateAllMirrorsWorker JID-e163149afff253a0718d3d4e INFO: start
2020-09-18T09:59:08.581Z 8 TID-gqwawidjc UpdateAllMirrorsWorker JID-e163149afff253a0718d3d4e INFO: done: 0.026 sec
2020-09-18T10:02:04.809Z 8 TID-oxf0ay1ss UpdateAllMirrorsWorker JID-47745f1778897cb0bb6ba20e INFO: start
2020-09-18T10:02:04.849Z 8 TID-oxf0ay1ss UpdateAllMirrorsWorker JID-47745f1778897cb0bb6ba20e INFO: done: 0.04 sec
2020-09-18T10:06:02.602Z 8 TID-gqwb24074 UpdateAllMirrorsWorker JID-3516f122abbac493da662a66 INFO: start
2020-09-18T10:06:02.686Z 8 TID-gqwamsnqk RepositoryUpdateMirrorWorker JID-e619695251a454447344eb93 INFO: start
Mirror update for xxx-import/xxx-cms started. Waiting duration: 0
2020-09-18T10:06:03.646Z 8 TID-gqwb24074 UpdateAllMirrorsWorker JID-3516f122abbac493da662a66 INFO: done: 1.044 sec
Mirror update for xxx-import/xxx-cms successfully finished. Update duration: 2}.
2020-09-18T10:06:04.238Z 8 TID-gqwamsnqk RepositoryUpdateMirrorWorker JID-e619695251a454447344eb93 INFO: done: 1.552 sec
2020-09-18T10:07:11.282Z 8 TID-gqwb24074 UpdateAllMirrorsWorker JID-a050780c28fe09705030aad3 INFO: start
Mirror job scheduled for xxx-import/xxx-scrapper with job ID 061e7a4aa8b9b51b639b83ed.
2020-09-18T10:07:11.351Z 8 TID-oxf0ay1ss RepositoryUpdateMirrorWorker JID-061e7a4aa8b9b51b639b83ed INFO: start
Mirror update for xxx-import/xxx-scrapper started. Waiting duration: 0
2020-09-18T10:07:12.318Z 8 TID-gqw9koaco UpdateAllMirrorsWorker JID-b550e05fcbcfc3ce2956db71 INFO: start
2020-09-18T10:07:12.323Z 8 TID-gqwb24074 UpdateAllMirrorsWorker JID-a050780c28fe09705030aad3 INFO: done: 1.041 sec
2020-09-18T10:07:12.348Z 8 TID-gqw9koaco UpdateAllMirrorsWorker JID-b550e05fcbcfc3ce2956db71 INFO: done: 0.03 sec
Mirror update for xxx-import/xxx-scrapper successfully finished. Update duration: 1}.
2020-09-18T10:07:12.627Z 8 TID-oxf0ay1ss RepositoryUpdateMirrorWorker JID-061e7a4aa8b9b51b639b83ed INFO: done: 1.277 sec
Mirror job scheduled for xxx-import/xxx-docs with job ID 67cabfa4656cea3692b66c7a.
Mirror job scheduled for xxx-import/xxx-users with job ID 8f58d2e1da7309ecb1d14690.
2020-09-18T10:09:01.766Z 8 TID-oxf7ic8og UpdateAllMirrorsWorker JID-cd76b9abeab2a1cf2efcaf21 INFO: start
2020-09-18T10:09:01.970Z 8 TID-oxf7ic8og UpdateAllMirrorsWorker JID-cd76b9abeab2a1cf2efcaf21 INFO: done: 0.204 sec
2020-09-18T10:13:13.266Z 8 TID-gqwb0mqnc RepositoryUpdateMirrorWorker JID-e6dad57d6953e73cb27e1fa8 INFO: start
Mirror update for xxx-import/xxx-poc started. Waiting duration: 0
Mirror update for xxx-import/xxx-poc successfully finished. Update duration: 1}.
2020-09-18T10:13:14.467Z 8 TID-gqwb0mqnc RepositoryUpdateMirrorWorker JID-e6dad57d6953e73cb27e1fa8 INFO: done: 1.201 sec
2020-09-18T10:17:02.287Z 8 TID-gqwb0mqnc UpdateAllMirrorsWorker JID-2450b361ac0e4c9cc484ee41 INFO: start
2020-09-18T10:17:03.329Z 8 TID-gqwb0mqnc UpdateAllMirrorsWorker JID-2450b361ac0e4c9cc484ee41 INFO: done: 1.042 sec
2020-09-18T10:19:17.288Z 8 TID-gqwb24074 UpdateAllMirrorsWorker JID-7015310b7519955a28464676 INFO: start
2020-09-18T10:19:17.317Z 8 TID-gqwb24074 UpdateAllMirrorsWorker JID-7015310b7519955a28464676 INFO: done: 0.029 sec
2020-09-18T10:21:10.124Z 8 TID-gqw9koaco RepositoryUpdateMirrorWorker JID-3c928f0ba32a0d1a99f0bf85 INFO: start
Mirror update for xxx-import/xxx-rt started. Waiting duration: 0
Mirror update for xxx-import/xxx-rt successfully finished. Update duration: 1}.
2020-09-18T10:21:11.918Z 8 TID-gqw9koaco RepositoryUpdateMirrorWorker JID-3c928f0ba32a0d1a99f0bf85 INFO: done: 1.794 sec
2020-09-18T10:22:03.151Z 8 TID-oxf2iw8kk UpdateAllMirrorsWorker JID-24ba5a49b4040adc6dd41094 INFO: start
2020-09-18T10:22:04.189Z 8 TID-oxfaugzjg UpdateAllMirrorsWorker JID-a708752d3ecb0e27417dcea0 INFO: start
2020-09-18T10:22:04.197Z 8 TID-oxf2iw8kk UpdateAllMirrorsWorker JID-24ba5a49b4040adc6dd41094 INFO: done: 1.046 sec
2020-09-18T10:22:04.224Z 8 TID-oxfaugzjg UpdateAllMirrorsWorker JID-a708752d3ecb0e27417dcea0 INFO: done: 0.035 sec
Also, a curl against the gitlab-exporter service yields:
curl gitlab-gitlab-exporter.gitlab.svc.cluster.local:9168/metrics | grep -i mirror
sidekiq_queue_size{name="cronjob:update_all_mirrors"} 0
sidekiq_queue_size{name="remote_mirror_notification"} 0
sidekiq_queue_size{name="repository_update_mirror"} 0
sidekiq_queue_size{name="repository_update_remote_mirror"} 0
sidekiq_queue_latency_seconds{name="cronjob:update_all_mirrors"} 0
sidekiq_queue_latency_seconds{name="remote_mirror_notification"} 0
sidekiq_queue_latency_seconds{name="repository_update_mirror"} 0
sidekiq_queue_latency_seconds{name="repository_update_remote_mirror"} 0
sidekiq_queue_paused{name="cronjob:update_all_mirrors"} 0
sidekiq_queue_paused{name="remote_mirror_notification"} 0
sidekiq_queue_paused{name="repository_update_mirror"} 0
sidekiq_queue_paused{name="repository_update_remote_mirror"} 0
gitlab_database_stat_table_seq_tup_read{table_name="project_mirror_data"} 105739514.0
gitlab_database_stat_table_seq_tup_read{table_name="remote_mirrors"} 33961.0
gitlab_database_stat_table_idx_tup_fetch{table_name="project_mirror_data"} 771202.0
gitlab_database_stat_table_idx_tup_fetch{table_name="remote_mirrors"} 288.0
gitlab_database_stat_table_n_tup_ins{table_name="project_mirror_data"} 828.0
gitlab_database_stat_table_n_tup_ins{table_name="remote_mirrors"} 34.0
gitlab_database_stat_table_n_tup_upd{table_name="project_mirror_data"} 668654.0
gitlab_database_stat_table_n_tup_upd{table_name="remote_mirrors"} 386.0
gitlab_database_stat_table_n_tup_del{table_name="project_mirror_data"} 750.0
gitlab_database_stat_table_n_tup_del{table_name="remote_mirrors"} 21.0
gitlab_database_stat_table_n_tup_hot_upd{table_name="project_mirror_data"} 178950.0
gitlab_database_stat_table_n_tup_hot_upd{table_name="remote_mirrors"} 321.0
gitlab_database_stat_table_n_dead_tup{table_name="project_mirror_data"} 64.0
gitlab_database_stat_table_n_dead_tup{table_name="remote_mirrors"} 25.0
gitlab_database_stat_table_seq_scan{table_name="project_mirror_data"} 1345844.0
gitlab_database_stat_table_seq_scan{table_name="remote_mirrors"} 5578.0
gitlab_database_rows{query_name="mirrors_ready_to_sync"} 0.0
gitlab_database_rows{query_name="mirrors_not_updated_recently"} 0.0
gitlab_database_rows{query_name="mirrors_updated_very_recently"} 0.0
gitlab_database_rows{query_name="mirrors_behind_schedule"} 0.0
gitlab_database_rows{query_name="mirrors_scheduled_or_started"} 0.0
gitlab_database_rows{query_name="mirrors_scheduled"} 0.0
gitlab_database_rows{query_name="mirrors_started"} 0.0
Edited by dgarcdu