DevEx: Test Governance (#116) · Epics · Developer Experience

DevEx: Test Governance

This epic is the SSOT for ongoing and planned Test Governance Team work. Please find below the work that is currently in-progress. ## Hours Worked | Epic | Category | DRI | Total Hours | |------|----------|-----|-------------| | [FY27Q2 Test Governance: Business as Usual](https://gitlab.com/groups/gitlab-org/quality/-/epics/391) | ~"infra-category::KTLO" | @jay_mccure @treagitlab | 0.0 | | [Modular Features Phase 2: Contract testing for Modular Features](https://gitlab.com/groups/gitlab-org/quality/-/epics/389) | ~"infra-category::Internal Innovation" | @mkomor1 | 0.0 | | [Modular Features Phase 2: Support Adoption](https://gitlab.com/groups/gitlab-org/quality/quality-engineering/-/epics/189) | ~"infra-category::Internal Innovation" | @jay_mccure @treagitlab | 0.0 | ## Project Work ### :hourglass: Work In Progress These epics are currently in progress | **Topic** | **Start Date** | **Target End Date** | **Summary** | |-----------|----------------|---------------------|-------------| | [FY27Q2 Test Governance: Business as Usual](https://gitlab.com/groups/gitlab-org/quality/-/epics/391) @jay_mccure @treagitlab (+0 participants) ~"group::test governance" | 2026-05-04 | 2026-07-31 | **2026-07-02**: :clock1: **total hours spent this week by all contributors**: 18 :tada: **achievements**: - MR reviews/backport investigations - Investigate cloud connector breakage in Omnibus tests - https://gitlab.com/gitlab-org/quality/test-failure-issues/-/work_items/31750#note_3524841900 - Investigate haml-lint job slowness https://gitlab.com/gitlab-org/gitlab/-/merge_requests/243605 - Review and responded to partner's image hardening test results _Copied from https://gitlab.com/groups/gitlab-org/quality/-/epics/391#note_3515179857_  **Nested Epics: 1** • https://gitlab.com/groups/gitlab-org/quality/-/epics/382+ | | [Modular Features Phase 2: Contract testing for Modular Features](https://gitlab.com/groups/gitlab-org/quality/-/epics/389) @mkomor1 (+0 participants) ~"group::test governance" | 2026-05-13 | 2026-06-19 | **2026-06-18**: :clock1: **total hours spent this week by all contributors**: 44 :tada: **achievements**: - https://gitlab.com/gitlab-org/quality/quality-engineering/modular-feature-testing/-/work_items/2+ - https://gitlab.com/gitlab-com/gl-infra/common-template-copier/-/merge_requests/673+ - https://gitlab.com/gitlab-com/gl-infra/common-ci-tasks/-/merge_requests/1602+ - https://gitlab.com/gitlab-org/quality/quality-engineering/modular-feature-testing/-/merge_requests/29+ - https://gitlab.com/gitlab-com/gl-infra/common-ci-tasks/-/merge_requests/1593+ - https://gitlab.com/gitlab-com/gl-infra/common-ci-tasks/-/merge_requests/1523+ :issue-blocked: **blockers**: - :arrow_forward: **next**: - Housekeeping items that does not affect the completion of this epic - https://gitlab.com/gitlab-org/quality/quality-engineering/modular-feature-testing/-/merge_requests/15+ - https://gitlab.com/gitlab-org/quality/quality-engineering/modular-feature-testing/-/work_items/7+s - This epic is complete with all direct child items resolved. Closing summary is provided below. Grand reviewer please close. _Copied from https://gitlab.com/groups/gitlab-org/quality/-/epics/389#note_3467361541_  | | [Modular Features Phase 2: Support Adoption](https://gitlab.com/groups/gitlab-org/quality/quality-engineering/-/epics/189) @jay_mccure @treagitlab (+0 participants) ~"group::test governance" | 2026-06-22 | 2026-07-24 | **2026-07-06**: _Copied from https://gitlab.com/groups/gitlab-org/quality/quality-engineering/-/epics/189#note_3525642200_ | ### :soon: Ready Linked epics that are ready to start | **Topic** | |-----------| | [Removing Pipeline DRI process](https://gitlab.com/groups/gitlab-org/quality/-/epics/371) ~"group::test governance" | ### :arrow_forward: Next These are the epics we will be focusing on next | **Topic** | **Target Start Date** | **Summary** | |-----------|-----------------------|-------------| | [Phase 2: Smarter Pipeline Triggering & Less Wasted Execution](https://gitlab.com/groups/gitlab-org/quality/-/epics/408) (+0 participants) ~"group::test governance" | | | | [Create an assistant for engineers to investigate failing tests](https://gitlab.com/groups/gitlab-org/quality/-/epics/292) (+0 participants) ~"group::test governance" | | | ### :stop_button: Triage Epics that are currently being triaged | **Topic** | **Summary** | |-----------|-------------| | [[Old] Test Governance: Business as usual](https://gitlab.com/groups/gitlab-org/quality/-/epics/142) ~"group::test governance" | **2026-02-26**: :clock1: **total hours spent this week by all contributors**: 34 :tada: **achievements**: - Supporting https://gitlab.com/gitlab-org/customers-gitlab-com/-/work_items/15383+ - Pipeline DRI 2/25 - 2/26 - Tiffany - Assisting flaky test investigation https://gitlab.com/gitlab-org/quality/test-failure-issues/-/work_items/28696 - Tiffany - 3 small MRs review & merge - Tiffany - Reviewed and provided feedback on a merge request the pre-emptively closed a modal, that would prevent an E2E failure when the feature is released, addressing pipeline failures that were preventing the merge train from being kicked off, helping to unblock the team's deployment @willmeek - Verified the correctness of the `GITLAB_QA_USER_AGENT` configuration to ensure proper test environment setup for customers-dot tests @willmeek - Support incident [Engineering is not receiving adequate response to red pipelines](https://app.incident.io/gitlab/incidents/7662) and other `master-broken` incidents @jay_mccure :issue-blocked: **blockers**: - :arrow_forward: **next**: - _Copied from https://gitlab.com/groups/gitlab-org/quality/-/epics/142#note_3114185688_ | | [Modular Features Phase 2: Test Strategy updates for Modular Features support](https://gitlab.com/groups/gitlab-org/quality/-/epics/392) ~"group::test governance" | | | [Modular Features Phase 2: Quality Guidelines for unsupported languages](https://gitlab.com/groups/gitlab-org/quality/-/epics/390) ~"group::test governance" | | | [Attach feature_category to frontend tests](https://gitlab.com/groups/gitlab-org/quality/-/epics/302) ~"group::test governance" | | | [Enable efficient database quality assessment](https://gitlab.com/groups/gitlab-org/quality/-/epics/301) ~"group::test governance" | | | [Reduce the defect escape rate by improving quality of top 2 groups with the highest bug escape rate](https://gitlab.com/groups/gitlab-org/quality/-/epics/300) ~"group::test governance" | | | [Quarantine Phase 2 - Enable Quarantine flow by ensuring all quarantined tests are deleted as low-value after 3 milestones](https://gitlab.com/groups/gitlab-org/quality/-/epics/277) ~"group::test governance" | | | [Phase 2 - Proactively monitor and alert on deployment blockage due to test failures](https://gitlab.com/groups/gitlab-org/quality/-/epics/272) ~"group::test governance" | | ### :white_check_mark: Completed Work Items that have been completed <details> | **Topic** | **Started** | **Ended** | **Summary** | |-----------| ------------| ----------| ------------ | | [Reduce E2E failure issue noise to simplify debugging](https://gitlab.com/groups/gitlab-org/quality/-/epics/176) ~"group::test governance" | 2025-05-05 | 2025-09-03 | **2025-08-20**: :tada: **achievements**: - I enabled the group similar functionality on Staging... :issue-blocked: **blockers**: - The passthrough to existing failure issue functionality if a grouped failure wasn't found didn't occur ( https://gitlab.com/gitlab-org/ruby/gems/gitlab_quality-test_tooling/-/issues/114 ) :arrow_forward: **next**: - I've already identified the small fix for this - https://gitlab.com/gitlab-org/ruby/gems/gitlab_quality-test_tooling/-/merge_requests/330 - then re-release test tooling _Copied from https://gitlab.com/groups/gitlab-org/quality/-/epics/176#note_2696553782_  | | [Improve feedback time on test pipelines via job parallelism and cleanup restructuring](https://gitlab.com/groups/gitlab-org/quality/-/epics/184) ~"group::test governance" | 2025-05-10 | 2025-07-11 | **2025-07-02**: :tada: **achievements**: * merged child pipeline pipeline-common!513 to speed up e2e test pipeline cleanup as a non-blocking job https://gitlab.com/gitlab-org/quality/quality-engineering/team-tasks/-/issues/3664. This represents a \~5-8 minute (\~25% gain in smoke sanity deployment QA pipelines) * a hot-fix was merged to correct the glob pattern today: https://gitlab.com/gitlab-org/quality/pipeline-common/-/merge_requests/523 * a follow-up issue was logged to consider a smarter cleanup fallback https://gitlab.com/gitlab-org/quality/quality-engineering/team-tasks/-/issues/3756 * tested the change alongside the \~10 minute (\~25-28%) time savings improvement in the `qa` section of the pipeline via added parallel test jobs https://gitlab.com/gitlab-org/quality/quality-engineering/team-tasks/-/issues/3654 * a partial revert of some parallel job was merged for ruling out staging environment and database instability https://gitlab.com/gitlab-org/quality/pipeline-common/-/merge_requests/526, reverting these gains where we're now limited by whichever longest running job finishes. We can re-evaluate adding back the parallel jobs in https://gitlab.com/gitlab-org/quality/quality-engineering/team-tasks/-/issues/3757 * alerted release-management, and released pipeline-common tag https://gitlab.com/gitlab-org/quality/quality-engineering/team-tasks/-/issues/3701 :arrow_forward: **next**: * Close epic - see closure status update from 2025-06-23 in https://gitlab.com/groups/gitlab-org/quality/-/epics/184#note_2580084392. * I accidentally didn't reply to the epic status thread last week and posted the update as a standalone comment, hence the gap from 2025-06-23 to today for this not to have closed last week. _Copied from https://gitlab.com/groups/gitlab-org/quality/-/epics/184#note_2597643746_  | | [Build test ownership across product teams with guides and learning materials](https://gitlab.com/groups/gitlab-org/quality/-/epics/179) ~"group::test governance" | 2025-05-26 | 2025-06-27 | **2025-06-25**: This epic can be labelled ~"workflow::complete" and closed. ([link to closure update](https://gitlab.com/groups/gitlab-org/quality/-/epics/179#status-2025-06-25)) | | [Design and document the test activities needed to support the monthly GitLab release](https://gitlab.com/groups/gitlab-org/quality/-/epics/174) ~"group::test governance" | 2025-06-02 | 2025-08-15 | **2025-08-13**: :tada: **achievements**: - This is done. We now have: - Defined backport testing responsibilities in handbook - https://handbook.gitlab.com/handbook/engineering/releases/backports/#backport-process-overview - Mechanism to update backport MR label to ~workflow::release-environment as soon as these changes are deployed to Release environment - https://gitlab.com/gitlab-org/gitlab/-/merge_requests/200861+ _Copied from https://gitlab.com/groups/gitlab-org/quality/-/epics/174#note_2684456136_  | | [Improve test resource cleanup for improved test stability](https://gitlab.com/groups/gitlab-org/quality/-/epics/208) ~"group::test governance" | 2025-07-21 | 2025-08-29 | **2025-08-27**: :tada: **achievements**: - Child test resource cleanup pipelines now verify deletions and alert on failure. They’re now up to par with scheduled resource deletion. This will allow us to be notified of deletion issues sooner and prevent resource pile up that can block deployments. - Within days of being implemented, this has caught a bug! https://gitlab.com/gitlab-org/gitlab/-/issues/565617 :issue-blocked: **blockers**: - :arrow_forward: **next**: - This can be closed! :tada: _Copied from https://gitlab.com/groups/gitlab-org/quality/-/epics/208#note_2711343751_  | | [Phase 1 - Reduce deployment blocked hours due to test failures by 20% per month](https://gitlab.com/groups/gitlab-org/quality/-/epics/206) ~"group::test governance" | 2025-08-19 | 2025-11-17 | **2025-11-12**: :tada: **achievements**: Grand Reviewers can close this epic as completed | | [Update GitLab test frameworks to support Project Studio](https://gitlab.com/groups/gitlab-org/quality/test-governance/-/epics/3) ~"group::test governance" | 2025-09-04 | 2025-11-21 | **2025-11-19**: **🎉 Closing Status - Project Studio Test Framework Update** The new UI is now default so we can close this epic. There is some cleanup work remaining in https://gitlab.com/gitlab-org/quality/quality-engineering/team-tasks/-/issues/4027+ which can only be completed once it is decided to remove the new UI toggle, I will move this issue to the BAU epic. Pain points encountered during this project have been recorded in https://gitlab.com/gitlab-org/quality/quality-engineering/team-tasks/-/issues/3884+ **Original Problem:** When Project Studio was introduced with its panel-based architecture, our test frameworks (both feature specs and E2E tests) needed significant updates to support the new UI. The challenge was multi-faceted: - Chrome needed to be upgraded from version 123 to 138 to properly handle CSS container queries used extensively in Project Studio - Both UI test frameworks required updates to work with the new interface - We needed to ensure engineers received feedback on their changes in MRs without breaking existing test coverage **Changes Made:** During this epic we: - **Upgraded Chrome across all test suites** from version 123 to 138 (a jump of 15 versions), enabling proper testing of CSS container queries - **Updated both test frameworks** (feature specs and E2E) to handle the new Project Studio UI, including the welcome banner that affected almost every test and flakiness introduced by a different version of Chrome :sob: - **Established blocking test pipelines** for Project Studio MRs, ensuring engineers get immediate feedback on their changes - **Configured E2E test pipelines** to run against Project Studio UI in staging, canary, and production environments - **Resolved runner infrastructure challenges** by working with the Runners Platform team to provision appropriate medium runners for the system test's (with Chrome 138) increased resource requirements **Impact:** - ✅ **Feature and E2E specs** against the new UI are now blocking in all MRs - ✅ **E2E test coverage** is now active against the new UI all environments (staging, canary, production) - ✅ **Test stability improved** through Chrome upgrade and systemic failure fixes - ✅ **Infrastructure optimized** with appropriate runner configurations for the new Chrome requirements Huge shoutout to @cwoolley-gitlab for the collaboration. Another huge shoutout to @dbickford & @tmaczukin for the runner infrastructure support! 🎊 _Copied from https://gitlab.com/groups/gitlab-org/quality/test-governance/-/epics/3#note_2897606618_  | | [Investigate how mandatory MFA effects test automation framework and tests](https://gitlab.com/groups/gitlab-org/quality/test-governance/-/epics/2) ~"group::test governance" | 2025-09-15 | 2026-01-30 | **2026-01-28**: :clock1: **total hours spent this week by all contributors**: 3 :tada: **achievements**: - @jroodnick investigation into GitLab's staging logs regarding [CDot E2E failures](https://gitlab.com/gitlab-org/customers-gitlab-com/-/work_items/15383) (moved back to [BAU](https://gitlab.com/groups/gitlab-org/quality/-/work_items/142)) :issue-blocked: **blockers**: - :arrow_forward: **next**: - @willmeek **Ready to Close** - The main `gitlab-org/gitlab` E2E test framework functions with mandatory MFA requirements. Our test suite bypasses MFA authentication and uses PAT logins instead of git credentials, ensuring seamless test execution in the MFA-enabled environment. _Copied from https://gitlab.com/groups/gitlab-org/quality/test-governance/-/epics/2#note_3039544701_  | | [Move test failure issues from gitlab-org/gitlab issue tracker to a different project](https://gitlab.com/groups/gitlab-org/quality/-/epics/222) ~"group::test governance" | 2025-10-02 | 2025-11-04 | **2025-10-15**: :tada: **achievements**: * Mappings were updated in the Test Tooling prior to the rename, such that issues continue to link to the correct pipelines and file URLs - https://gitlab.com/gitlab-org/ruby/gems/gitlab_quality-test_tooling/-/merge_requests/349 * Project was renamed from `e2e-test-issues` to https://gitlab.com/gitlab-org/quality/test-failure-issues , it was observed that existing E2E test updates continued to function * Investigation carried out as to what might be affected, any changes that might need made (eg. analytics) * Update made to CI and quarantine script at https://gitlab.com/gitlab-org/gitlab/-/merge_requests/208349 * Observed that feature/integration spec issues are being raised and updated at https://gitlab.com/gitlab-org/quality/test-failure-issues * Announced in `s_developer-experience` slack channel https://gitlab.slack.com/archives/C07TWBRER7H/p1760455442760099 :issue-blocked: **blockers**: * :arrow_forward: **next**: * Move existing open issues from `gitlab-org/gitlab` * Continue to observe and update any breaking automations/integrations/analytics * Check for documentation that needs updating * Nadya to announce change in `engineering-fyi` _Copied from https://gitlab.com/groups/gitlab-org/quality/-/epics/222#note_2821264584_  | | [Re-distribute E2E test suites between test pipelines to remove duplications](https://gitlab.com/groups/gitlab-org/quality/quality-engineering/-/epics/163) ~"group::test governance" | 2025-10-27 | 2026-01-16 | **2025-12-24**: :clock1: **total hours spent this week by all contributors**: 4 :tada: **achievements**: - The original problem: - We have multiple same test suites run in these 3 test pipelines: `e2e:test-on-omnibus`, `e2e:test-on-gdk`, `e2e:test-on-cng` - A brief description of the changes made - We have added new configurations to [CNG Orchestrator](https://gitlab.com/gitlab-org/gitlab/-/tree/master/qa/gems/gitlab-cng?ref_type=heads) to enable more test suites to run in `e2e:test-on-cng` pipeline: GitHub Oauth, Registry, Relative-url - Followed these changes, we removed the above mentioned suites together with `Intance` and `Gitaly-transactions` suites from `e2e:test-on-omnibus` pipeline. - Information about the impact of the project now that it is completed: - We have reduced number of jobs in `e2e:test-on-omnibus` from [111 jobs](https://gitlab.com/gitlab-org/gitlab/-/pipelines/2217020164) down to [72 jobs](https://gitlab.com/gitlab-org/gitlab/-/pipelines/2230302342) - i.e: 39 redundant jobs in both stable branch MR pipelines as well as master branch 2-hourly scheduled pipelines. ![Screenshot_2026-01-09_at_11.07.56_AM](/uploads/64ed6a0eea6e4fbab7f2211da10a364b/Screenshot_2026-01-09_at_11.07.56_AM.png){width=900 height=202} - We can see that the average runtime in minutes between the 3 pipelines have balanced out to be roughly 200 minutes or less per week since December up to now. Charts can be found in [Snowflakes](https://app.snowflake.com/ys68254/gitlab/#/dx-e2e-test-dashboards-d8JKf0WUW) - Grand Reviewers can close this epic as done :arrow_forward: **next**: - We can't yet remove `gdk-instance` jobs because they are currently used by ~"group::development analytics" to [collect test coverage data](https://gitlab.com/gitlab-org/quality/analytics/team/-/work_items/339). There is an epic opened to remove this in the future https://gitlab.com/groups/gitlab-org/quality/test-governance/-/epics/5+ _Copied from https://gitlab.com/groups/gitlab-org/quality/quality-engineering/-/epics/163#note_2972581484_  **Nested Epics: 1** • https://gitlab.com/groups/gitlab-org/quality/quality-engineering/-/epics/165+ | | [Phase 1: Reduce Functional test flakiness by updating Quarantined Test Process](https://gitlab.com/groups/gitlab-org/quality/-/epics/195) ~"group::test governance" | 2025-11-04 | 2026-02-06 | **2026-02-04**: :clock1: **total hours spent this week by all contributors**: 16 [Epic closure message](https://gitlab.com/groups/gitlab-org/quality/-/epics/195#note_3059938610) as it did not display correctly in parent epic. Epic can be closed, thank you! 🙏 _Copied from https://gitlab.com/groups/gitlab-org/quality/-/epics/195#note_3056325085_  **Nested Epics: 4** • https://gitlab.com/groups/gitlab-org/quality/-/epics/275+ • https://gitlab.com/groups/gitlab-org/quality/-/epics/258+ **2026-01-21**: :clock1: **total hours spent this week by all contributors**: 2 :tada: **achievements**: - As per last item - https://gitlab.com/gitlab-org/quality/quality-engineering/team-tasks/-/issues/4057 - Opened MR to update public-facing docs (gitlab-org/gitlab!219562) to consolidate quarantine documentation by pointing to the handbook as single source of truth, such that the process can continue to be detailed and maintained in a single location. :issue-blocked: **blockers**: - :arrow_forward: **next**: - Address minor review feedback on MR and get it merged. As this is the last item, this epic issue can be marked complete and closed. _Copied from https://gitlab.com/groups/gitlab-org/quality/-/epics/258#note_3021855461_  • https://gitlab.com/groups/gitlab-org/quality/-/epics/257+ **2025-12-17**: • https://gitlab.com/groups/gitlab-org/quality/quality-engineering/-/epics/171+ | | [Adjust CI Pipeline configuration to reduce costs and merge duration](https://gitlab.com/groups/gitlab-org/quality/-/epics/288) ~"group::test governance" | 2025-11-10 | 2026-02-24 | **2026-02-19**: | | [Removal of Project Studio-specific test infrastructure](https://gitlab.com/groups/gitlab-org/quality/-/epics/315) ~"group::test governance" | 2025-11-17 | 2026-01-16 | **2026-01-07**: This epic was created retrospectively to represent the work involved in removing the ~"Project Studio"-specific test infrastructure, and can now be **closed**. This was to capture the reduction in CI minutes usage as suggested in a previous DevEx grand review. Parallel testing of both UIs was turned off on **December 23rd** with https://gitlab.com/gitlab-org/gitlab/-/merge_requests/215882+ Looking at CI usage minutes from the month prior to switching the tests off, we can predict how much we will approximately reduce CI minutes used in future months: | | [Support DAP on Dedicated](https://gitlab.com/groups/gitlab-org/quality/-/epics/290) ~"group::test governance" | 2025-11-25 | 2026-01-30 | **2026-01-28**: :clock1: **total hours spent this week by all contributors**: 0 :tada: **achievements**: - :issue-blocked: **blockers**: - :arrow_forward: **next**: - **Ready to close** - I've completed the initial scope of this epic. I performed current test analysis, identified coverage gaps, and worked with the Dedicated SREs to establish the test pipeline for DAP on Dedicated. The DAP team will conduct manual testing to reach GA (~Feb 23rd), with framework enhancements addressed reactively as needed post-GA (see gitlab-org/quality/quality-engineering/team-tasks#4054 ) _Copied from https://gitlab.com/groups/gitlab-org/quality/-/epics/290#note_3039544858_  | | [Insights on time spent on external Merge Requests](https://gitlab.com/groups/gitlab-org/quality/-/epics/295) ~"group::test governance" | 2025-12-08 | 2026-02-05 | **2026-02-04**: :clock1: **total hours spent this week by all contributors**: 6 :tada: **Analysis**: - If we don't account for the two weeks of holiday(which were both 0hrs), the running average for hours spent by the Test Governance team on external merge requests is 6h per week with a high of 10h and a low of 2h. - This is ready to be closed _Copied from https://gitlab.com/groups/gitlab-org/quality/-/epics/295#note_3056325189_  | | [Optimize Test Pipeline Performance Without Compromising Quality](https://gitlab.com/groups/gitlab-org/quality/-/epics/282) ~"group::test governance" | 2026-02-01 | 2026-06-08 | **2026-05-28**: **Duo Test Selection** closes out with strong results: ~3M system-test job-minutes saved per month since go-live, with a 66% average reduction in system tests run across tier-2 pipelines and no widening of the tier-2/tier-3 failure gap. :clock1: **total hours spent this week by all contributors**: 24 :tada: **achievements**: - [**Phase 2: Duo Test Selection — Implement in Tier-2 Pipelines**](https://gitlab.com/groups/gitlab-org/quality/-/work_items/377) *(closing)* - Lowered confidence threshold from 80% to 70% to expand Duo selection to more pipelines, saving additional compute time ([#4371](https://gitlab.com/gitlab-org/quality/quality-engineering/team-tasks/-/work_items/4371)). - Removed `experiment` label from job, marking the feature as production-ready ([#4405](https://gitlab.com/gitlab-org/quality/quality-engineering/team-tasks/-/work_items/4405)). - Merged fix for over-sharding on predictive system test pipelines ([#4442](https://gitlab.com/gitlab-org/quality/quality-engineering/team-tasks/-/work_items/4442)): system shards reduced by 38% (78 → 48), saving ~194 CI-minutes per pipeline run. - Compute savings calculated and documented in [#4377](https://gitlab.com/gitlab-org/quality/quality-engineering/team-tasks/-/work_items/4377). - **Closing summary:** https://gitlab.com/groups/gitlab-org/quality/-/work_items/377#note_3405146510 _Copied from https://gitlab.com/groups/gitlab-org/quality/-/epics/282#note_3392210406_  **Nested Epics: 5** • https://gitlab.com/groups/gitlab-org/quality/-/epics/377+ **2026-05-28**: :clock1: **total hours spent this week by all contributors**: 24 :tada: **achievements**: - Decreased the confidence threshold to 70% (from 80%) to run Duo Test Selections against more pipelines (saving more compute time) https://gitlab.com/gitlab-org/quality/quality-engineering/team-tasks/-/work_items/4371+ - Removed `experiment` label from job https://gitlab.com/gitlab-org/quality/quality-engineering/team-tasks/-/work_items/4405+ - Fixed over-sharding on predictive system test pipelines https://gitlab.com/gitlab-org/quality/quality-engineering/team-tasks/-/work_items/4442+: raises the system test job target from 600s to 900s, reducing system shards by 38% (78 → 48) and saving ~194 CI-minutes per pipeline - Calculated compute savings in https://gitlab.com/gitlab-org/quality/quality-engineering/team-tasks/-/work_items/4377 :arrow_forward: **next**: - close off epic _Copied from https://gitlab.com/groups/gitlab-org/quality/-/epics/377#note_3392210472_  • https://gitlab.com/groups/gitlab-org/quality/-/epics/337+ **2026-04-23**: • https://gitlab.com/groups/gitlab-org/quality/-/epics/335+ **2026-04-30**: :clock1: **total hours spent this week by all contributors**: 4 :tada: **achievements**: - We no longer build the qa-image for as-if-FOSS pipelines https://gitlab.com/gitlab-org/quality/quality-engineering/team-tasks/-/work_items/4392+ (E2E tests don't run in this pipeline) - Removed mattermost E2E test job (saving ~15 minutes of an `e2e` runner each pipeline), due to it being deprecated in Omnibus package. https://gitlab.com/gitlab-org/quality/quality-engineering/team-tasks/-/work_items/4436+ _Copied from https://gitlab.com/groups/gitlab-org/quality/-/epics/335#note_3301698119_  • https://gitlab.com/groups/gitlab-org/quality/-/epics/329+ **2026-03-19**: :clock1: **total hours spent this week by all contributors**: 28 :tada: **achievements**: - Built dashboards tracking recall %, bail-out rate, selected spec counts, runtime projections and missed failure drill-downs https://app.snowflake.com/ys68254/gitlab/#/dx-duo-predictive-tests-dc8lJBn0j - Analysed 7 days of shadow mode data (954 valid pipelines, 679 with Duo predictions) - Recall metric (currently 17.52%) is understated due to master-broken tests and flaky specs contaminating the failure set — needs filtering before drawing firm conclusions (for example, an [investigation](https://gitlab.com/gitlab-org/quality/quality-engineering/team-tasks/-/work_items/4190#note_3172869541) into 30 failing pipelines in one day showed a very high percentage of master-broken/flaky tests) :arrow_forward: **next**: - Investigate master-broken test filtering to get a cleaner recall signal - Investigate how Duo test selection can complement current strategies - Document outcomes of POC and next step options - Assess readiness to be used in tier-2 pipelines _Copied from https://gitlab.com/groups/gitlab-org/quality/-/epics/329#note_3172499041_  • https://gitlab.com/groups/gitlab-org/quality/-/epics/330+ **2026-04-17**: :tada: achievements: * Completed small runner (2 vCPU) POC runs with vmstat instrumentation, definitively confirming CPU saturation as the bottleneck for multi-worker on small runners * Posted full findings with pipeline links, vmstat data, and per-worker throughput analysis to #4315 * Updated descriptions of #4315 and #330 with complete POC results, breakeven analysis, and recommendations | | [Modular Features Phase 1: Basic test tooling setup for Golang](https://gitlab.com/groups/gitlab-org/quality/-/epics/357) ~"group::test governance" | 2026-03-02 | 2026-05-21 | **2026-05-14**: 🕐 **Total hours spent this week by all contributors:** 30 🎉 **Achievements:** - Completed a key Phase 1 milestone for the [Modular Features testing epic](https://gitlab.com/groups/gitlab-org/quality/-/work_items/357) by getting the `common-ci-tasks` integration [!1484](https://gitlab.com/gitlab-com/gl-infra/common-ci-tasks/-/merge_requests/1484) reviewed and merged. This matters because it establishes the scalable delivery path for the Go testing framework into consuming modular feature projects, moving the work from a reference implementation toward something teams can adopt through the standard platform workflow. It directly advances the epic goal of a standardized testing framework that modular feature teams can adopt with minimal setup effort, and keeps Phase 1 broadly aligned with the expected timescale, offset by a couple of weeks of emergency leave. 🚫 **Blockers:** None. ▶️ **Next:** - Close out the remaining Phase 1 follow-up work, including the lighter-weight `common-template-copier` [!602](https://gitlab.com/gitlab-com/gl-infra/common-template-copier/-/merge_requests/602), then move into Phase 2, including contract testing. _Copied from https://gitlab.com/groups/gitlab-org/quality/-/epics/357#note_3346433433_  | | [FY27Q1 Test Governance: Business as Usual](https://gitlab.com/groups/gitlab-org/quality/-/epics/361) ~"group::test governance" | 2026-03-05 | 2026-05-07 | **2026-04-30**: :clock1: **total hours spent this week by all contributors**: 65 :tada: **achievements**: - MR Reviews - Investigating bug that's failing test cleanup - Fixing bug with renovate bot - Update pipeline-triage scheduling to allow no DRI weeks - Created [MR](https://gitlab.com/gitlab-org/quality/engineering-productivity-infrastructure/-/merge_requests/1346) to fix Terraform provisioning for a couple environment variables in CDot project - Pipeline triage DRI @jay_mccure - Created [MR](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/233883) to support E2E tests from planned helm chart removal of bundled PostgreSQL, Redis, MinIO :issue-blocked: **blockers**: - :arrow_forward: **next**: - _Copied from https://gitlab.com/groups/gitlab-org/quality/-/epics/361#note_3301698418_  | | [Understanding unit test coverage to enable Project studio](https://gitlab.com/groups/gitlab-org/quality/test-governance/-/epics/4) | | 2025-10-07 | | </details> ### :rotating_light: Epics that need attention These linked epics are not in the correct state or missing a workflow label <details> | **Topic** | **Links** | **Reason** | |-----------|-----------|-------------| | [Duo Workflow - Testing(Experiment Release)](https://gitlab.com/groups/gitlab-org/-/epics/14158) @amyphillips (+0) group::ai framework [DEPRECATED] | Epic has no workflow label but is closed | | [Simplify user experience of E2E test orchestration tools](https://gitlab.com/groups/gitlab-org/quality/-/epics/106) (+1) group::test governance | Labeling problem, epic has no workflow label | | [Move on from stable counterpart model](https://gitlab.com/groups/gitlab-org/quality/-/epics/140) @amyphillips (+0) group::test governance | Labeling problem, epic has no workflow label | | [Editor extensions test tooling and test enablement](https://gitlab.com/groups/gitlab-org/-/epics/16891) @jay_mccure (+0) group::test governance | Labeling problem, epic has no workflow label | | [Design and launch new Test Governance Request For Help process](https://gitlab.com/groups/gitlab-org/quality/-/epics/156) @amyphillips @willmeek (+0) group::test governance | Labeling problem, epic has no workflow label | | [Eliminate SET On-call: Phase 2: Live Environment Pipeline Frequency Reduction](https://gitlab.com/groups/gitlab-org/-/epics/16167) @treagitlab (+0) group::test governance | Labeling problem, epic has no workflow label | | [Create testing training resources to enable company-wide testing](https://gitlab.com/groups/gitlab-org/quality/-/epics/151) (+0) group::test governance | Labeling problem, epic has no workflow label | | [Review our automated tests and plan future improvements](https://gitlab.com/groups/gitlab-org/quality/-/epics/173) @chloeliu (+3) group::test governance | Labeling problem, epic has no workflow label | | [Enable feature_category instead of product_group in e2e tests' metadata](https://gitlab.com/groups/gitlab-org/quality/-/epics/197) @tim_beauchamp (+1) group::test governance | Labeling problem, epic has no workflow label | | [Standardized Testing Solution for Modular Features - Phases](https://gitlab.com/groups/gitlab-org/quality/-/epics/386) @ndenisenko (+0) group::test governance | Labeling problem, epic has no workflow label | | [Phase 2: Duo Test Selection - implement in tier-2 pipelines](https://gitlab.com/groups/gitlab-org/quality/-/epics/372) (+0) | Epic has ~"workflow-infra::Proposal" but is closed | | [CNG E2E test pipeline stability issues and flakiness](https://gitlab.com/groups/gitlab-org/quality/-/epics/307) (+0) | Labeling problem, epic has no workflow label | | [Support transition from Redis to Valkey](https://gitlab.com/groups/gitlab-org/quality/-/epics/297) (+0) group::test governance | Epic has ~"workflow-infra::Triage" but is closed | | [Help development teams to identify and design more effective tests to improve quality](https://gitlab.com/groups/gitlab-org/quality/test-governance/-/epics/1) (+0) group::test governance | Labeling problem, epic has no workflow label | | [Test improvements to reduce maintenance and effort](https://gitlab.com/groups/gitlab-org/quality/-/epics/187) (+0) group::test governance | Labeling problem, epic has no workflow label | | [Plan stage: audit E2E tests by removing and identifying tests to be migrated to spec/features](https://gitlab.com/groups/gitlab-org/quality/-/epics/157) @dchevalier2 (+0) group::test governance | Labeling problem, epic has no workflow label | | [Verify stage: audit E2E tests by removing and identifying tests to be migrated to spec/features](https://gitlab.com/groups/gitlab-org/quality/-/epics/158) @treagitlab (+0) group::test governance | Labeling problem, epic has no workflow label | | [Govern stage: audit E2E tests by removing and identifying tests to be migrated to spec/features](https://gitlab.com/groups/gitlab-org/quality/-/epics/160) (+0) group::test governance | Labeling problem, epic has no workflow label | | [Fulfillment & Growth stage: audit E2E tests by removing and identifying tests to be migrated to spec/features](https://gitlab.com/groups/gitlab-org/quality/-/epics/161) @jroodnick (+0) group::test governance | Labeling problem, epic has no workflow label | | [Stabilize gitaly-related E2E test jobs](https://gitlab.com/groups/gitlab-org/quality/-/epics/180) @jay_mccure (+0) group::test governance | Labeling problem, epic has no workflow label | | [Shift down identified e2e tests to features spec level](https://gitlab.com/groups/gitlab-org/quality/-/epics/170) (+0) group::test governance | Labeling problem, epic has no workflow label | | [Create a minimal testing layer to act as a quality safety net](https://gitlab.com/groups/gitlab-org/quality/-/epics/177) @amyphillips (+0) group::test governance | Epic has no workflow label but is closed | | [Test Governance - Create a Single Source of Truth for Testing Best Practices](https://gitlab.com/groups/gitlab-org/quality/quality-engineering/-/epics/121) (+0) group::test governance | Labeling problem, epic has no workflow label | | [Introduce a minimal set of metrics to assess E2E test effectiveness](https://gitlab.com/groups/gitlab-org/quality/quality-engineering/-/epics/132) (+0) group::test governance | Labeling problem, epic has no workflow label | | [Improve balance of our test pyramid by removing E2E tests identifying tests to be migrated to a lower level](https://gitlab.com/groups/gitlab-org/quality/-/epics/153) @chloeliu @jay_mccure (+0) group::test governance | Labeling problem, epic has no workflow label | | [Update CustomersDot E2E Test Documentation to Help Facilitate Developer Involvement](https://gitlab.com/groups/gitlab-org/-/epics/13268) @jroodnick (+0) group::test governance | Labeling problem, epic has no workflow label | | [Enhance Fulfillment E2E Test Coverage Across GitLab and CustomersDot MRs](https://gitlab.com/groups/gitlab-org/-/epics/16663) (+0) group::test governance | Labeling problem, epic has no workflow label | | [Application Security Testing: Test planning and implementation for high-priority features](https://gitlab.com/groups/gitlab-org/quality/-/epics/143) @willmeek (+0) group::test governance | Labeling problem, epic has no workflow label | | [Enable Engineering teams to own E2E tests on-call](https://gitlab.com/groups/gitlab-org/quality/-/epics/130) @amyphillips (+0) group::test governance | Labeling problem, epic has no workflow label | </details>

epic