Skip to content

[E2E] Add logging regarding sidekiq queue status if E2E fails

What does this MR do and why?

[E2E] Add logging regarding sidekiq queue status if E2E fails

We occassionally run into flaky test failures which can be quite difficult to reproduce. I suspect that some of these may be due to requests being eineued f but for various reasons it may not process in sidekiq. In our CI suite there should be minimal items in the queue, so perhaps if notice during test execution that the queue is building up, we can determine that a test failure was due to this.

How to set up and validate locally

  1. Create a Personal Access token with API only scope: export GITLAB_QA_ACCESS_TOKEN={PERSONAL_ACCESS_TOKEN}

  2. Stop sidekiq: gdk stop rails-background-jobs

  3. Run Spec and observe it fails.
    QA_HOSTNAME=http://gdk.test:3000 QA_LOG_LEVEL=debug CI=true CI_PROJECT_NAME=localhost bundle exec rspec qa/specs/features/browser_ui/3_create/merge_request/create_merge_request_from_push_notification_spec.rb:39

  4. Observe new log message after spec fails [Dec 11 2023 13:58:51 GMT (QA Tests)] DEBUG -- sidekiq/compound_metrics: "{\"queues\":{\"default\":{\"backlog\":11,\"latency\":65},\"mailers\":{\"backlog\":1,\"latency\":65}},\"processes\":[],\"jobs\":{\"processed\":122372,\"failed\":1,\"enqueued\":12,\"dead\":0}}

  5. Restart sidekiq and monitor queue drains gdk start rails-background-jobs curl --location 'http://gdk.test:3000/api/v4/sidekiq/compound_metrics' --header 'PRIVATE-TOKEN: {PERSONAL_ACCESS_TOKEN}'

  6. Run Spec and observe it passes.
    QA_HOSTNAME=http://gdk.test:3000 QA_LOG_LEVEL=debug CI=true CI_PROJECT_NAME=localhost bundle exec rspec qa/specs/features/browser_ui/3_create/merge_request/create_merge_request_from_push_notification_spec.rb:39

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by John McDonnell

Merge request reports