Skip to content

Mechanism to add rspec meta to specs

Sanad Liaquat requested to merge tti-shl-test-meta into main

What does this MR do and why?

Describe in detail what your merge request does and why.

As part of gitlab-org/quality/quality-engineering/team-tasks#1918 (closed) (OKR), we want to automate promotion to reliable bucket and quarantining reliable test that are no longer reliable.

In gitlab-org/gitlab!139742 (merged), we created json files as output of script that runs the reliable report. The json files contain specs that need to be quarantined or promoted to reliable.

This MR implements code that uses those json files and create MRs for quarantining unstable reliable specs and promoting specs to reliable.

Notes:

  • This MR depends on !128 (merged). Once that MR is merged, the target branch of this MR should be changed to main.
  • The code in this MR does not decide which tests to quarantine or promote to reliable. It just takes a json file and acts on it.
  • There is a limitation in the quarantining code. It currently does not detect if a spec is already in quarantine.

Credits

Thanks to @ddavison for his work on the confiner project. Some code in this MR takes inspiration from that project.

Screenshots or screen recordings

These are strongly recommended to assist reviewers and reduce the time to merge your change.

How to set up and validate locally

For example MRs already created as part of testing, please see: https://gitlab.com/sliaquat/gitlab/-/merge_requests?scope=all&state=all

For testing quarantining of unstable reliable specs:

QA_TEST_DRI_HANDLE='<your_gitlab_handle>' CI_SLACK_WEBHOOK_URL="<available_in_1p>" SLACK_QA_CHANNEL="qa-notifications-test-channel" QA_LOG_LEVEL='debug' exe/update-test-meta  --project "sliaquat/gitlab" --token "<your_personal_access_token>" --unstable_specs_file "path/to/unstable_specs.json"

Notes:

  • QA_TEST_DRI_HANDLE is for testing only to avoid pinging other team members. When not provided, the code will use the roulette to ping the appropriate stage SET.
  • SLACK_QA_CHANNEL set here is for testing only. When not provided, the message will be send to test-platform-maintainers Slack channel.
  • In --project, provide a GitLab project where you would like to operate on and create MRs. It is best to use your own fork of the GitLab project to avoid creating unnecessary MRs. A point to note is that MRs will not have all the labels if your fork doesn't have them.
  • For --unstable_specs_file, you can use this file as an example: unstable_specs.json

For testing promoting stable specs to reliable:

QA_TEST_DRI_HANDLE='<your_gitlab_handle>' QA_LOG_LEVEL='debug' exe/update-test-meta  --project "sliaquat/gitlab" --token "<your_personal_access_token>" --stable_specs_file "path/to/stable_specs.json"

Notes:

  • QA_TEST_DRI_HANDLE is for testing only to avoid pinging other team members. When not provided, the code will use the roulette to ping the appropriate stage SET.
  • In --project, provide a GitLab project where you would like to operate on and create MRs. It is best to use your own fork of the GitLab project to avoid creating unnecessary MRs. A point to note is that MRs will not have all the labels if your fork doesn't have them.
  • In --stable_specs_file, you can use this file as an example: stable_specs.json

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Sanad Liaquat

Merge request reports