Investigate how triage-ops works, which projects, how it can automate labelling of our backlog One question to answer is how many of the product projects are included: https://gitlab.com/gitlab-data/analytics/-/blob/master/transform/snowflake-dbt/seeds/seed_engineering/projects_part_of_product.csv Side note: Triage only runs on issues, not epics (the functionality doesn't exist for epics at the gem level). ## Triage ops - Handbook page: https://handbook.gitlab.com/handbook/engineering/infrastructure/engineering-productivity/triage-operations/ - Project: https://gitlab.com/gitlab-org/quality/triage-ops - Ownership: shared, primarily Development Analytics - Policies: https://gitlab.com/gitlab-org/quality/triage-ops/-/tree/master/policies/generated ### Path lists - Groups: https://gitlab.com/gitlab-org/quality/triage-ops/-/blob/master/triage/triage.rb#L9 - Projects: https://gitlab.com/gitlab-org/quality/triage-ops/-/blob/master/triage/triage/event.rb#L72 ### Team and type labels - Team inference (auto-label of section/stage based on group): https://gitlab.com/gitlab-org/quality/triage-ops/-/blob/master/triage/processor/team_label_inference.rb#L28 - Applies to group: https://gitlab.com/gitlab-org/quality/triage-ops/-/blob/master/triage/processor/team_label_inference.rb#L16 - "group": com, org, components, community: https://gitlab.com/gitlab-org/quality/triage-ops/-/blob/master/triage/triage/event.rb#L304 - Type label inference: https://gitlab.com/gitlab-org/quality/triage-ops/-/blob/master/triage/processor/label_inference.rb#L94 - runs in org group and runbooks project ### AI triager Maintainer: `@leetickett-gitlab` - AI helper `AiTriageHelper.new` https://gitlab.com/gitlab-org/quality/triage-ops/-/blob/master/lib/ai_triage_helper.rb - Function including `issue_triage_ai_comment` defined in: https://gitlab.com/gitlab-org/quality/triage-ops/-/blob/master/plugins/issue_triage_helper.rb#L9 - Triage rule: https://gitlab.com/gitlab-org/quality/triage-ops/-/blob/master/policies/stages/report/untriaged-issues.yml - If triaged function: https://gitlab.com/gitlab-org/quality/triage-ops/-/blob/master/lib/untriaged_helper.rb#L67 ; which excludes authored by bot, feature flag, “special” labels, type ignore, has type + group already, `automation:ml` label - schedule in `pipeline-schedules.yml` - `.gitlab/ci/report.yml` maps to `untriaged-issues` - does not apply labels if below confidence: https://gitlab.com/gitlab-community/gitlab-org/quality/triage-ops/-/blob/master/lib/ai_triage_helper.rb#L35 Changes made: 1. MR to run on existing issues: https://gitlab.com/gitlab-org/quality/triage-ops/-/merge_requests/3440 ## Key findings 1. Group inference (auto add stage/section based on group) runs in gitlab-org, gitlab-com, and components - Should cover all projects that are “part of the product” 2. Type inference (add type based on subtype) runs in gitlab-org group and runbooks project only (if I’m reading the code correctly) - Obviously, this doesn’t cover all the projects, though most of them 3. AI triage only runs on gitlab (monorepo) project, and does not label if \< 80% confidence ### Recommendations 1. Extend type inference to gitlab-com and components 2. Increase the number of issues processed at a time. The main thing is we don’t want to hit rate limits. 3. Consider other, cheaper models to use for the AI triager. 4. If the backlog run goes well, extend AI triage, which adds group and (sub)type if missing to gitlab-org, and relevant components and gitlab-com projects. 5. We can consider lowering the confidence by 5 or 10% to catch more issues. Feedback should be gathered from Product and Eng on how effective labelling has been before adjusting.