Sharing CI Backend work across Verify groups that impacts SaaS customers
Problem to Solve
In light of the growing concern where ~"group::continuous integration" continues to have a number of escalations as highlighted in this Google Doc (internal), we continue to struggle to have capacity to address other critical issues in our backend backlog, or schedule any kind of work in our milestone.
Another consideration is given too much on ~"group::continuous integration"'s plate, and we run the risk of burning out our Backend team as they've been fully allocated to Engineering-driven escalations since February.
Proposal
There has been some ideas surfaced already to distribute some of the higher priority issues, such as sharing the CI bug backlog with other groups in the Ops section, and leveraging GraphQL expertise on other teams to help unblock our FE team on CI.
I'd like to refine these down to a single list of CI issues, where we identify a backlog of ~"technical debt" ~bug security ~performance issues impacting operational stability of gitlab.com given our SaaS first direction.
Request
- The ask is to have ~"group::testing" or grouppipeline authoring schedule some issues (e.g. 1-2 per milestone), or consider them as Filler items that BE engineers can help with once their Deliverable work is complete. This would work typically prioritized as part of the regular milestone for the Backend CI team. It would be up to the PM/EM to determine how CI backend issues might fit in their list of priorities.
- grouprunner is not included in this distribution given that their Backend ICs are well-versed in Golang, not Rails
Examples of this type of work might include:
- priority1 ~performance or security or infradev issues (or even priority2)
- follow ups from rapid action that should be addressed in the next 2-3 milestones (e.g. impacting db performance on .com)
- other backend issues as highlighted in this issue (DRIs have been contributing towards): https://gitlab.com/gitlab-org/ci-cd/continuous-integration/-/issues/78
These are just ideas for now, and I'm happy to discuss alternatives, or put together a more defined list.
Tagging @jreporter @darbyfrey for initial review prior to sharing this more broadly.