Add Reliability Metrics to Github Importer Grafana Dashboard

Current Github Grafana Dashboard

We currently have a simple grafana dashboard that counts some object importing and limit rates

Proposed changed

The current dashboard leverages from a global counter of imported objects (https://gitlab.com/gitlab-org/gitlab/blob/master/app/workers/concerns/gitlab/github_import/object_importer.rb#L50). Using the introduction of the number of fetched and imported objects for each import log (!64256 (merged)) as example, a similar global counter for the fetched objects could be introduced.

Having the expected number of objects (fetched number of object) and the imported number of objects enables us to calculate a success rate which would be imported/fetched.

Ideally we could have a global success rate and by object type success rates, like:

- Global success rate: sum(all imported objects)/sum(all fetched objects)
- Merge Requests imported success rate: number of imported merge requests/number of fetched merge requests
- Issues imported success rate: number of imported issues/number of fetched issues
- ... 

TODO

  • Count fetched and imported objects globally !65700 (merged)
  • Count objects imported in batches (Gitlab::GithubImport::BulkImporting) !65773 (merged)
    • Enable the feature flag created import_redis_increment_by on staging
    • Enable the feature flag created import_redis_increment_by on production
  • Update dashboard to use the new counters (gitlab-com/runbooks!3793 (merged), gitlab-com/runbooks!3817 (merged))
    • https://dashboards.gitlab.net/d/importers-github-importer/importers-github-importer-test
  • Share the dashboard with the group in slack
  • Add the dashboard link to the dev documentation
    • !68554 (merged)
Edited Aug 24, 2021 by Kassio Borges
Assignee Loading
Time tracking Loading