Review the Optimize categories and identify opportunities to consolidate.
Problems to solve
Depth over breadth - The product team has shifted to a focus on depth in core product areas. We have barely scratched the surface with the value that can be created in Value Stream Management and DevOps Reports. These two categories will lay the foundation for a high-level, centralized view of the end-to-end operations for product organizations, allowing the target personas of executive, VP, and director to quickly identify problems and evaluate the value of investments and deliverables. There is tremendous opportunity to be a market leader in these two categories. To be successful, we need to focus on depth, and narrow down the personas that we are attempting to service.
Not easy to get an overall view - Many of the analytics in GitLab today are specific to a stage. Customers don't get an overall view to present to senior leadership. groupoptimize is uniquely positioned to provide an overall view because they are not aligned to a specific DevOps stage. It makes sense for each stage to build out metrics specific to their area, especially because they have the subject matter expertise on which metrics are useful, while groupoptimize focuses on the full picture.
Large number of categories for a small team - groupoptimize is currently responsible for five product categories. This is a large number of categories for a small team (2 x FE, 2 x BE (plus one on leave for the first half of 2021). We have already determined that we only have capacity to focus on two of the categories in 2021: Value Stream Management and DevOps Reports.
Lack of vision - The three other categories (planning analytics, code analytics, and insights) do not have a detailed vision or maturity plan and would be easy to remove, absorb into Value Stream Management or DevOps Reports, or hand off to a stage-specific team.
Reduce the categories to two by doing the following:
Fold the Code Analytics category into Value Stream Management and DevOps Reports
The code analytics category is described as a deep dive into the Code stage of Value Stream Analytics to find bottlenecks. Category:Value Stream Management already provides high-level insight into the time that workflow items spend in the code stage relative to other stages of the software development life cycle and allows users to drill in and identify bottlenecks. Value Stream Analytics will link to other key parts of the UI so that users can dive deeper. Additionally, DevOps Reports will show usage metrics for features in the Create stage and how usage correlates to performance.
There is an open discussion about whether Code Analytics moves to Create or is absorbed into VSM and DevOps Reports. This depends on whether Create has a vision and customer demand for more metrics.
Similar to Code Analytics, the planning analytics category is described as a deep dive into the Plan stage of Value Stream Analytics to find bottlenecks. Category:Value Stream Management provides high-level insight into the time spent on planning, and a sortable list of issues in the plan stage so that users can identify lagging work. devopsplan has subject matter expertise on the detailed analytics specific to the Plan stage that would be most valuable to users, as outlined by @gweaver.
Fold Insights into Value Stream Management and DevOps Reports
Insights does not have a direction page. The maturity plan and direction are unknown. The Insights page in the GitLab UI provides a breakdown of the types of issues that are being worked on and ability to create custom dashboards. The roadmap for Category:Value Stream Management and DevOps Reports includes some customizability and a breakdown of work by issue type. As these two categories continue to evolve, they can incorporate the value provided through the Insights page.
Keep Value Stream Management and DevOps Reports and drive maturity in these categories
@david What are your thoughts on removing these three categories? I found a process in the handbook for adding categories but not for removing them. Are there any special approvals or processes we need to follow?
@ogolowinski I'd really like to get this issue resolved. We don't have the capacity to maintain five categories. Are you aware of a process for removing categories?
@adawar I'm copying your questions from the vision review doc so I can address them here. Thanks for asking two very important questions.
Anoop: Category removal - Is this because we need surface area on the product, reduce tech-debt and maintenance or because we don’t think it is useful?
The Planning Analytics and Code Analytics categories were created at the same time last year. There is no vision or plan to go with them and no features were built. There would be no material impact of removing these categories besides just not showing them as categories on our website and not having the overhead of maintaining/creating direction pages and maturity plans.
Planning Analytics - Has a very minimal Direction page with no plan. It does not have a maturity page. The Direction page states "Planning Analytics seeks to deep-dive into the Plan stage of the DevOps lifecycle, by providing insight into your instance's planning activity." but there is no proposal of how to do that. @gweaver is already building some metrics in Plan and we can work with him on how to surface key planning metrics in VSA as part of Category:Value Stream Management
Code Analytics - This category has an epic but it is mostly empty with no JTBDs or proposal. It is described as a deep dive into Create metrics, but similar to Planning Analytics I think we should give Create the freedom to work on their own stage-specific metrics that we can roll up and display in VSA, especially considering the small size of the Optimize team.
Insights - there is no direction page for this category so I don't know what the vision was for this. The feature was originally created by Quality team to build dashboards they need. I am working with Mek to see how we can provide key metrics through VSA. The groupoptimize team's mission is to provide insights through VSA and DevOps Reports so I don't think we need to build a separate category around this.
Anoop: Category removal - what is our estimated adoption of the categories we propose to remove?
Planning analytics and Code analytics - no features were ever created so there is no adoption to consider.
The Insights category is tied to the Insights feature. In April we had 2,239 unique visits to project-level Insights and 634 visits to group-level Insights. We don't currently plan to remove this page but we're also not actively improving it. As we mature Category:Value Stream Management and DevOps Reports there will be less need for Insights.
@ljlane So my understanding is that your proposal is to have 2 categories under groupoptimize
DevOps Reports
Value Stream Management
If this is correct, I feel like that is ok, but I feel there is a missing category for "General Analytics" (feel free to rename) which theoretically consolidates all the analytics available at GitLAb today under one roof, which I feel is missing
Just from the left pane these are available:
but there are also code quality and security (and future alert and IR) analytics elsewhere. I think groupoptimize should own this, without having to be the group that actively adds each metric.
@ljlane@ogolowinski we desperately need to make progress on Planning Analytics. The category is very much in demand from a customer standpoint. Whether we kill the category or move it to Plan, specific things we need in Plan:
Velocity, volatility, and resource capacity planning
Evolving insights to be more expressive and provide more in-depth customizable reporting.
I don't have strong opinions on who owns these or where they live so long as it we add support for them in the not-so-distant future (~12 mo. time horizon).
@gweaver Please do not let this or us hold you back. groupoptimize is focusing on DevOps reports and VSM - please feel free to work on whatever you need to move your stage forward.
I think that ultimately groupoptimize will be the place groups all the analytics together, but each stage will be responsible for instrumenting their analytics in a contribution like architect similar to compliance and even the merge request widget.
@ogolowinski - @gweaver has a clear vision and understands the requirements for the engineering manager/project manager/product manager user. groupoptimize is focused on executive/VP/director dashboards. If the Plan Analytics category continues to exist I think we should transfer it to @gweaver on the understanding that we'll collaborate to highlight some key planning metrics in VSA. IMO that makes more sense than groupoptimize being responsible for maintaining direction and maturity plans for Plan Analytics.
Some teams are building analytics directly into their part of the product, such as https://gitlab.com/groups/gitlab-org/-/security/dashboard. IMO this is a great approach because users can access analytics directly from within the context that they are already in, instead of navigating away from Security & Compliance, for example. In doing this, a user specifically interested in compliance or planning, because that's relevant to their role in their organization, can stay within the area of the product where they usually spend the most time. It then becomes the responsibility of groupoptimize to provide an end-to-end view of the development workflow for those user personas that are focused on overall productivity, investment, ROI, customer value, etc.
Over time, we could consolidate the analytics pages in the left nav in a VSA or DevOps Adoption view, or migrate them to other parts of the UI that provide relevant context so they are more easily discoverable. For example, code review analytics might be more discoverable if it lived within Merge Requests. In VSA we can display a key Review metric such as average review time, and link to the Code Review table from VSA. Similarly, CI/CD analytics in the CI/CD left nav. To use @djensen's words, we'd take a hub and spoke approach where VSA is the hub and links to spokes that are within the appropriate area of the UI given the context.
@ljlane You can either move the Planning Analytics category into Project Management or completely remove it. Either way, when we start building analytics we will certainly collaborate closely with you. My working strategy is to implement metrics in such a way that they are location agnostic as we need to integrate them into Boards, other reports, and eventually user-configurable metrics dashboards.
It would also be extraordinarily useful if there were GraphQL APIs for all of groupoptimize metrics reports. For example, if a public VSA API existed, we could create custom value stream reports for Board workflows and easily integrate the data from the VSA reports into the context of a Board.
I think there may be some value in defining a standard architecture pattern for building and exposing various metrics so it is easy for groupoptimize to also roll them up where appropriate.
Is there an issue describing the standard architecture pattern? I know you've mentioned this before but I'm not sure exactly what is needed or if we already have one.
@ljlane I don't think there is one anywhere right now. This was a general idea long ago -- &3653 (closed) with the idea of teams contributing metrics to the library so we could easily compose dashboards and what not -- &3707 (closed)
@gweaver@ljlane what if we had a namespace in the public API that we used as our "library"? These endpoints could return responses with (as much as possible) a standardized format. This would give everyone a clear place to contribute new metrics. Would that solve for all the aspects we want to solve for?
@ljlane -Thank you for the diligent followup. This makes sense to me.
No action expected -but just a suggestion.
Be ambitious in your long term vision and plan. It is ok to have categories that we would like to work on in the future --and we can even call out the issues that we want worked on -in the hopes that they would encourage community contribution.
Then we can call out the categories we do plan to focus on in the near term to improve their maturity.
Be ambitious in your long term vision and plan. It is ok to have categories that we would like to work on in the future
This is a great point @adawar There is a lot we can do to build a centralized view into end-to-end operations that will highlight problem areas and show the value of inputs and outputs. I am consistently hearing that this is a gap. Focusing the team on this vision gives us an opportunity to think about new categories that align with this vision.
we can even call out the issues that we want worked on -in the hopes that they would encourage community contribution
Also a great point. I'll think about how we can do more of this.
@sarahwaldner I've been reviewing the categories that groupoptimize is responsible for and how they align with our vision as a group. There is a category called Category:Code Analytics which was created last year but was never worked on. I'm proposing that groupoptimize drop this category and focus on categories that provide an overview of the entire software development flow rather than detailed, stage-specific metrics. Before we do, do you have any opinions on the need to keep it or a desire for devopscreate to take this over?
On a slightly related note, Analytics in the project-level left nav has links to repository, code review, and merge request analytics. As I commented above, these analytics might be more discoverable if they are in the Repository and Merge request sections of the nav. I can open a separate issue for that discussion.
Category:Code Analytics seems like something the devopscreate stage can absorb as we are the experts in what analytics our users would want to track.
@phikai There is a proposal by @ljlane to rehome Category:Code Analytics under Create. After reviewing the existing direction page, this strikes me as an excellent opportunity for us to add value to higher tiers in Code Review. Can you please review and let me know what you think?
I understand you have a lot of other pressing initiatives ongoing right now. If we were to move the category to groupcode review, we could %Backlog it immediately while we develop our higher tier roadmap.
Thank you for your thoughts @sarahwaldner. If we go that path, we will work closely with @phikai to incorporate key metrics for Create into the Value Stream Analytics and DevOps Adoption dashboards.
@sarahwaldner@ljlane I'm personally not a huge fan of moving this under devopscreate or groupcode review because I think this creates inefficiencies in understanding users as well as disjointed user experiences. We'd essentially end up with 2 or 3 groups creating "analytics" experiences vs. having a single group own the user experience and information presented in our analytics.
Thinking from a PM perspective, this doesn't make much sense either as it seems unlikely that we'd enter a conversation about code review and end up in metrics. But it's likely you'd enter a conversation talking about high level metrics and get probed about more detailed metrics and questions that users would like to answer.
I'm fine if this is the plan - but I don't know why we'd keep the category on ice in our group vs. keep the category on ice in it's current spot. And if both groups are going to keep it on ice, should it even exist?
I am curious how @gweaver has thought about this since I know that the Plan stage is absorbing Project Management metrics in their purview.
Gabe - Kai makes some good points here:
I think this creates inefficiencies in understanding users as well as disjointed user experiences. We'd essentially end up with 2 or 3 groups creating "analytics" experiences vs. having a single group own the user experience and information presented in our analytics.
@phikaigroupoptimize is providing a high-level overview of the end-to-end flow of value streams. We are targeting executive, VP, and Director users. We are working on a unified view that represents each stage using a key set of metrics, as well as the investment and return on investment that organizations are getting from their value streams. This unified view, along with the discussion about a standard pattern, should help with your concern about a disjointed user experience. We don't have the capacity or subject matter expertise to deep dive into detailed metrics for every stage in the software development lifecycle. Instead, we would like to link out to each stage for users who are interested in exploring more details using a hub and spoke model where VSA is the hub. For now, VSA could link to existing analytics for the Create stage, such as Analytics>Code Review.
I am curious how @gweaver has thought about this since I know that the Plan stage is absorbing Project Management metrics in their purview.
@sarahwaldner Gabe has been thinking about Plan analytics for a long time and has gathered quite a lot of customer feedback on the needs here. There is considerable demand for plan analytics from project managers, product managers, and engineering managers. If additional analytics is not an immediate need or priority for Create, then we could remove the category but continue to support high-level Create metrics through Category:Value Stream Management and DevOps Reports.
It's your call @phikai and @sarahwaldner. I'll create an MR to action your decision.
However, the list of issues in the %Backlog for Category:Code Analytics, does demonstrate interest and have a variety of ideas potentially worth exploring.
@ljlane I understand that you are trying to scope down what your team is focused on and you can do so with or without Create taking ownership of this category.
@phikai I would encourage you to review and consider the existing content for this category. It does not need to remain as a defined category - it could exist as a different category or part of one. I want to make sure that we aren't dismissing an opportunity
@sarahwaldner - My concern isn't with whether the category should exist (I think it should), it's about where's the most appropriate place for it to live.
IMO - the more we can keep all the analytics experiences in a single group the better it will ultimately be for our users. One consistent vision/direction/voice and ultimately experience is better than dividing that up across groups.
IMO - the more we can keep all the analytics experiences in a single group the better it will ultimately be for our users. One consistent vision/direction/voice and ultimately experience is better than dividing that up across groups.
Thank you for your perspective.
@ljlane If this category is not "rehomed" to Create, what is your plan for the direction page and category as a whole?
@sarahwaldner We will absorb the category into Value Stream Management and DevOps Reports. We'll display key analytics related to code creation and review in VSA and the DevOps Reports. For pages that already exist under the Analytics left nav, such as Analytics>Code Review, we'll either move these metrics into the VSA/DevOps Reports pages or we'll leave them where they are but link to them from VSA and DevOps Reports so that we're providing a smoother deep dive experience and making it easier to find analytics of interest. We're also looking at an API that will make it easier for other teams to contribute metrics specific to their stage. What we won't be focused on in the next 12 months is building new metrics that are specific to the Create stage.
I agree that the analytics categories can be consolidated into VSM so Optimize will focus on only 2 categories in the future.
DevOps Reports
VSM
Each stage/group shall be responsible for creating their own analytics and VSM will be central hub that will make it easier for users to find them. So the groupoptimize should be aware of new metrics and analytics added to the product but should not be the ones responsible for implementation.
There is a proposal to move planning analytics to devopsplan.
Correct.
Insights, VSM & code analytics will all be consolidated into Value stream management
There is still an open discussion about Code Analytics. This category will either be moved to Create if they have a specific vision for analytics, or it will be absorbed into VSM (for time analytics) and DevOps Reports (for usage analytics).
For Insights, yes, the proposal is to incorporate that into the Category:Value Stream Management where we show insights into value streams both through VSA and the Insights page.
DevOps Reports remains the same
Correct. DevOps Reports will show usage analytics for each stage, and correlations between usage and performance.
please revise the proposal in the issue so that it is clearer
I added a comment about the options for Code Analytics. Is there anything else that is unclear in the proposal?
In my discovery on Code Search, Code Analytics does come up. As we start on code search I would like to keep looking at how we can enable analytics for code.
@ljlane One thing that was not clear to me is if there are any features that are coming to devopsplan with the transfer of Planning Analytics? It wasn't clear to me based on the direction page or MR. Also, is there a label for that category? I couldn't find it .
@ljlane There is one report that will come over as part of this transition. That's what I understood based on this MR !67879 (merged) . Let me know if that's what you had in mind with the transition Planning Analytics to groupproject management .
Can you help me find open issues related to this report so we can move them to groupproject management ? I wasn't able to identify labels that we should use.
@mushakov I don't think there was a Category:Planning Analytics label, so I went ahead and created that. We may want to create an issue analytics feature label to cover the one report view.