CI/CD categories refresh

Problem to Solve

FY20 category planning at start of year was overly optimistic, need to reassess based on hiring and Q1 delivery progress. In general, priorities have evolved and the plan for the year needs a refresh both in terms of an evolving market, concerns with customer sentiment around the maturity of flagship categories, and a renewed focus across the board on not maximizing new categories.

Proposal

This is being addressed in three parts - reduction in number of categories, adjustment of target maturities/dates, and general features cleanup.

Categories to Remove

Removing categories reduces diffusion of PM focus since categories now come with a minimum amount of ongoing maintenance. By removing categories that we were unlikely to bring to or past minimal status, we achieve the goal of both allowing PMs to focus on the more important things while removing potentially misleading content.

• Incremental Rollout (Release): This category has been part of continuous delivery, but was primarily created to serve as a marketing signal. The proposal here is to remove the category completely, but if the marketing signal remains important we can consider renaming the CD category to something like "Continuous and Incremental Delivery". In my own experience and research, incremental rollout tooling is an important term but is primarily an abstract concept related to feature flags and other CD techniques. We are encapsulating this idea now in out overall CI/CD strategy with the Progressive Delivery theme.

• Compatibility Testing (Verify): There is a minimal market here, and compared to bringing other more broadly used Verify testing packages to a higher level of quality, along with more general efforts such as building dashboarding across all of quality management. It's potentially interesting one day, but at the moment there are just 2 issues with upvotes. One has six and one has ten upvotes, but nothing in this category has sparked any significant customer interest.

• Linux Package Repository (Package): This is too niche for to go after for now. In comparison with other Package priorities (RubyGems, Maven, NPM) it is not valuable to expend focus on this category. There are immediate concerns with functionality of NPM and Maven that must be urgently addressed, and this will take the remainder of the year to deliver. Customer engagement has been pretty minimal, there are 4 issues with between 3 and 7 upvotes each. There are better opportunities for us here.

• Helm Package Repository (Package): This is an exciting category, but still one that suffers from a comparative lack of urgency compared to other Package priorities. This is one we'd likely look to re-add next year, based on a market assessment. We have had modest engagement via issues (2 issues with the headliner being https://gitlab.com/gitlab-org/gitlab-ce/issues/35884, having 23 upvotes.) Compared to Linux this is more interesting, but more in the "something to keep an eye on" phase than something we want to get to complete or even minimal as a stake in the ground yet.

Instead of Helm or Linux (which the CAB also corroborated were not immediately interesting), we are considering adding other categories based on demonstrated customer demand. Research is ongoing (and these will be added in a separate MR), but strong candidates so far are the following with significantly more interest than either of the ones being removed:

• .net: 86 upvotes
• PHP: 55 upvotes

Adjustments to Target Maturities & Dates

The new state can be viewed in summary in the review app at https://jlenny-master-patch-73426.about-src.gitlab.com/handbook/product/categories/maturity/. In general here, the goal of this edit is to focus on fewer things (taking advantage of the category cancellations above) in order to bring more strategic items to a higher level of maturity more quickly. Q1 performance did not meet expected goals, so this makes category planning a bit more realistic in terms of what the team can deliver. Especially considering Release and Package will share resources until any engineers are able to be hired for Package.

• Verify
  • Code Quality: Changes target date of complete from May 22 to July 22
  • Performance Testing: Changes target of viable on May 22 to complete on October 22
  • System Testing: Removes year end target of complete
  • Usability Testing: Changes target of complete on May 22 to viable on July 22. The focus will be on https://gitlab.com/gitlab-org/gitlab-ee/issues/10761 as a way to bring the category to viable without further investment needed at this point.
• Package
  • Container Registry: Changes target date of complete from May 22 to October 22
  • Maven Repository: Changes target of viable on May 22 to complete on July 22
  • NPM Repository: Changes target date of complete from May 22 to October 22
• Release
  • Release Orchestration: Changes target date of complete from May 22 to July 22
  • Pages: Removes year end target of loveable. Pages would need to be at that maturity to compete with pure-play hosting solutions, but not for its purpose within GitLab.
  • Review Apps: Removes year end target of loveable. Focus on synergies with Feature Flags/Release Orchestration in context of Progressive Delivery rather than general Review App improvements.
  • Feature Flags: Sets target date of complete to July 22
  • Release Governance: Sets target date of viable to July 22

This keeps things ambitious, but hopefully more realistic. It's difficult to make accurate assessments here given the unknown timelines for hiring Package team members but we don't want to plan for the worst case.

A separate problem for some of these categories (particularly Package ones) is that there is no epic or other resource specifically defining what viable, complete, etc. mean for these categories. Once this MR is approved and merged, defining those will be the next task.

Feature Accuracy

Various incorrect feature data was also reviewed and corrected - this was mostly various older features assigned to wrong categories in features.yml.