Project 'gitlab-org/quality/team-tasks' was moved to 'gitlab-org/quality/quality-engineering/team-tasks'. Please update any links and bookmarks that may still have the old path.
Average CE/EE `master` end-to-end test suite execution duration per month
Description: Measures the average duration of our full QA/end-to-end test suite to accelerate cycle time of merge requests and release process.
The measure should be in master branch after changes to tests have been merged. So:
Nightlies
Staging
I am a bit unclear how the parallelize test runs are being used in these 2 areas if they are using the same mechanism. I would assume yes. If we know the ci job names and branchs it should be relatively easy to pull into Periscope.
Link issues together to show that they're related or that one is blocking others.
Learn more.
Activity
Sort or filter
Newest first
Oldest first
Show all activity
Show comments only
Show history only
Mek Stittrichanged title from Quality Engineering KPI - Average CE/EE end-to-end test suite execution duration per month to Quality Engineering KPI - Average CE/EE {+master +}end-to-end test suite execution duration per month
changed title from Quality Engineering KPI - Average CE/EE end-to-end test suite execution duration per month to Quality Engineering KPI - Average CE/EE {+master +}end-to-end test suite execution duration per month
I think these are 2 different population of tests and we can't just sum and average them. The long poll should be Nightlies since that is the true full suite with orchestrated tests e.g. Ldap and etc
@kwiebers: Out of curiosity where do you see the duration data come from? Is the package-and-qa manual jobs in the ce/ee pipelines or breaking up the ee/ce jobs from the semi-daily runs at: https://gitlab.com/gitlab-org/quality/nightly/pipeline_schedules
Still getting oriented to some of the QA processes and curious where this would come from.
@meks: Good question! package-and-qa runs on a feature branch and we don't have control over in-coming changes or instability being introduced. The measure here should be after it has merged into master
So
Nightlies
Staging
I would like to extend this to the duration of the tests in the deployer pipeline as well that probably too much. In theory, if the tests in staging and nightlies are performant the execution in other areas should also benefit.
Kyle Wieberschanged title from Quality Engineering KPI - Average CE/EE master end-to-end test suite execution duration per month to Average CE/EE master end-to-end test suite execution duration per month
changed title from Quality Engineering KPI - Average CE/EE master end-to-end test suite execution duration per month to Average CE/EE master end-to-end test suite execution duration per month
We discussed how review-build-cng + review-deploy would be similar to the trigger:package job in package-and-qa and how the review-qa-smoke + review-qa-all is similar (but still different) from the trigger:QA job
We also decided that looking at staging Sanity and Full QA pipeline durations over time was helpful beyond what was reviewed. These jobs are on the path to production. We found that it was hard to filter between Sanity/Full pipelines but the stages allowed us to filter.