We need to start addressing 2 key challenges enterprise GitLab customers face:
GitLab ROI & Value - Backward-looking justification - Understanding and demonstrating the ROI of GitLab by understanding project, group & instance-level activity (see GitHub Enterprise).
Transformation progress & benchmarking - Forward-looking guidance - Understanding progress of DevOps transformation and how their company compares to others in similar industries, markets, etc. Getting advice on how they can further their progress.
We plan to start by tackling 1, with 2 as the end goal.
Ultimately, we are helping our users understand and communicate how their DevOps effort metrics are impacting business value and product management (customer) value. To do this we must help them understand how to define, measure, and track business value from different points of view — and ensure that developers deliver meaningful value for the business and its customers.
Issues created: Group >> Issue >> list - this would work if issue lists had a date range picker component
MRs created: Group >> MR >> list - this would work if issue lists had a date range picker component
Pipelines run: I don't think there is a view for this one yet
Releases: nor this
Security scans: nor this
Vulnerabilities: Group >> Security - this actually lines up coincidentally with our pre-selected date range
- Link has a proper destination (I think)
In future iterations, we can show these numbers over time and call out specific features that are not being used that are part of that instance's tier (e.g. use CI).
This page may contain information related to upcoming products, features and functionality.
It is important to note that the information presented is for informational purposes only, so please do not rely on the information for purchasing or planning purposes.
Just like with all projects, the items mentioned on the page are subject to change or delay, and the development, release, and timing of any products, features, or functionality remain at the sole discretion of GitLab Inc.
This feature will list exactly what the "group details" view does. The only new content in this view are the boxes/panels above the table, which sum the data for the group.
Since instance level analytics, needed data at the group level too, the data collection won't change.
Basically, what were going to show in the bottom table, i.e. group level analytics, is what we will show on the group page.
@jeremy Could you please clarify if instance level analytics are going to be needed at all?
Is this issue a replacement for the previous issue or an additional feature?
Are we going to offer date ranges eventually? This changes the data storage from counters to time series data - these are vastly different data amounts for GitLab.com and can help us decide the data storage backend for it
How current does the data have to be? That can help us decide how often and how to run the counts.
@aakriti.gupta let's 1) generally assume date ranges are desirable future enhancements for Analytics features and 2) not let that influence our decisions for this MVC. I feel like we have enough work on our hands validating we are able to query and store the latest counts, without adding the complexity of storing historical counts.
How current does the data have to be?
It's going to be the last 90 days, so will need to be refreshed daily.
Could you please confirm what tiers this feature will be available for? For now I am going with, minimum EE - starter plan, and Bronze on GitLab.com, like Code Review and Contribution Analytics.
Since instance level analytics, needed data at the group level too, the data collection won't change.
@aakriti.gupta just to make 100% sure we're all on the same page: I think instance-level activity only needed data for top-level groups, whereas this feature needs data for sub-groups too. So I think the data collection needs to be more granular, and the data storage will be higher volume as a result.
I'm still working on estimating how much data this will generate on gitlab.com.
Some questions we want to answer regarding data storage for this MVC:
Is the data needed in time series or only overall counters?
While Postgres could store overall counts that don’t add as much data, it cannot support gitlab.com scale of time-series data.
What is the scale of the data? Or, for which users is this feature going to exist?
What is the granularity of data reporting?
What we decided in a call (Dan, Ben, Adam and me, today):
We need to reduce the scope of the MVC to see what we need to build to validate the problem
We need to decide now if the medium term and long term feature is going to be viable - that has to be part of the MVC
Some ideas for the MVC:
Run daily batch jobs to count data for gitlab-org group and roll this feature out only for this one group, to validate the feature
Test our the performance of these queries and time taken
Estimate how much time and storage space queries for the whole of gitlab.com are going to take
Finalize if we will in future need a time series db, so that we are not doing work that is going to be useless later
Consider exposing data readable by external services
One thing that we've discussed in the past has been allowing customers to get structured data out of GitLab so they're able to create their own dashboards. We took a nice step forward with structured application logging (thanks, aakriti), but that's something that only self-managed instances can really take advantage of. How should we approach this for groups on GitLab.com, I wonder?
I like the idea of providing the users with the data and allowing them to make their own dashboards, choose how and for how long they store data etc.
For self-managed instances, one way is to expose Prometheus readable data on /group-name/-/metrics that users' own Prometheus can scrape. They can then setup their own dashboards. Setting up one more Prometheus instance is not much work.
For gitlab.com, we can setup our own Cortex which is a long term storage for Prometheus and offer a dashboard with a limited set of querying options, e.g. date range selection.
Thanks to @bjk-gitlab for helping out with these ideas!
I need to run some numbers on the queries we'll be running for the MVC.
But since so far, we are only re-using some queries that are used elsewhere in the sidebar (total issues and MRs counts), it is safe to say that they won't bring GitLab down.
The API call will be made async, so the page load should not be badly affected either.
@aakriti.gupta Do you plan on implementing this as an API endpoint that can be consumed by the FE? If so, can we agree on an API spec? I'm happy to work on the FE part for this.
@jeremy If I understand correctly, then we're going to add the 8 metrics to the existing group overview page?
So, we'll need a FE issue for this as well which I can create. Can you please confirm that this is required? If so, then I can sync with @aakriti.gupta on the API spec.
@wortschi While implementing the API endpoint and adding more metrics to it I realized that the endpoint might take very long to return the results of all the queries which are very long anyway. I might have to split the data into separate endpoint: /analytics/group_activity/{issues_count, merge_requests_count, recent_members_count}
Just wanted to give you a heads up since you're probably quite far with the development.
Please keep me posted about the metrics each endpoint is going to provide. As for the MVC in 12.9, maybe we can start with just showing Issues created and MRs created as initially discussed, given that both metrics will be provided by the same endpoint? In the next iteration we could add more metrics via additional endpoints?
I am waiting on BE reviews/approval on two MRs (queries and API endpoints). Once these are merged, we can release this feature's MVC through the API. Hopefully within 12.9 :)
Do we show the total/absolute count of the issues, etc. from all sub-groups and projects of the group, even if some of those are not accessible to the viewing user?
What happens when a group member views these stats for a group with several private sub-groups and projects, that they're not a part of?
If we store counts for a group, we cannot filter this data later, by user access levels.
Some queries could potentially be run on the fly, e.g. sum of issues, sum of merge requests. In fact we do these already, to show a count in the sidebar.
But others, e.g. total pipelines run for the last 90 days, would be too large and would need to be run through async workers.
I've consulted with @aakriti.gupta on this issue - thanks a lot for the conversation. Since we're still determining the best approach to how we store analytics data, we decided the best approach in this MVC was to focus on metrics we can query for synchronously when loading the group overview page.
My suggestion is that we display the following metrics:
Count of new members added to the group within the 90 day period
Count of created merge requests in the 90 day period
Count of created issues in the 90 day period
If possible:
Pipelines run in the 90 day period
Deploys in the 90 day period (I think we have this on the Value Stream Analytics page)
Active members in the 90 day activity (group members with an event in the group, subgroups, or projects)
If this isn't possible, consider "active issues" or "active MRs", which would be all issues/MRs that were updated in the 90 day time period
We can display these in a single container, like we do on the VSA page:
Since we're doing these queries live, that handles several questions above. One that isn't covered:
Do we show the total/absolute count of the issues, etc. from all sub-groups and projects of the group, even if some of those are not accessible to the viewing user?
I think we should only count what the user is able to access.
@npost: Also note that Matej did a group overview redesign a while ago that may be helpful. I'm more than fine with the simple container we're going with in the proposal.
I have quite a few MRs already in review or merged.
But there is more work to be done to split the API endpoint into separate ones, one each for each statistic.
This is because the queries are quite long and we would need to make one query per api call, instead of all queries in one call.
I am pausing work on this MVC for another 2-3 days, to catch up with Performance Training, that took a backseat in order to get this MVC out in this milestone.
I understand from my chat with Jeremy, that it is ok to prioritize the training over this for now. We can release this feature in the next milestone, in case we can't do it in 12.9 itself.
At this point, I can imagine us releasing the API endpoints in 12.9, but I am not sure if the API endpoints will be done well in time for FE to use them.
The new endpoints will be /analytics/group_activity/issues_count and /analytics/group_activity/merge_requests_count instead of /analytics/group_activity.
@aakriti.gupta we need docs for the feature at some point, so should probably go ahead and create a PR now. (We should mark it as dependent on the frontend MR so it doesn't get released too early.) Are you up for preparing that?
@jshackelford@jeremy@djensen The API MR is ready to be merged today. If it makes it to master before the release is cut, should block it by feature flag?
(When the feature flag is not set at all, it is treated as being set to true for all groups.)
I don't think we're releasing the FE in this release
So if it's a new API with no changes to existing endpoints, I don't know if there's a reason to feature flag it
It would just not be used until the FE goes in, I would think
The controller checks for a feature flag, but for this release it will neither be enabled nor disabled, which amounts to the same functionality as it being enabled for all.
Okay ... For now, the FE skeleton is already merged but even with an enabled feature flag we won't show anything to the user since neither the UI nor the API calls have been merged yet.
@wortschi Do I understand it correctly, that we're not releasing the FE in 12.9?
Based on the comment in #207164 (comment 303881864) I did update the milestone for the FE to 12.10. It's already the last day for 12.9 so I doubt that FE is going to get merged today given that the API isn't merged either.
@djensen@jeremy@dennis It isn't clear to me how close we are to being able to deliver this in 12.10 if we were to pursue that. What is left for a complete FE, BE solution?
@jshackelford It looks like the MR which introduces the API endpoints got merged a few hours ago, see !26460 (merged)
The FE portion is still WIP and was on hold until now. Now that the API is available we can continue on the FE part but this will not make it into 12.9 (I believe today is the last day for the current milestone).
@jshackelford Do we need to enable the feature for a few groups first for solution validation/testing, or is it fine to enable it for all of production in one go?
@jshackelford as I am testing this feature out, I have found some interesting permission cases.
Could you please confirm what an anonymous or signed in user, who does not have access to a public group, sees?
Is the recent activity container visible to them or no?
The MRs, issues, members are visible to this user. But since this feature is for users to see their own use of GitLab, it doesn't make much sense to show it to users who are not part of the group. On the other hand, this data is already visible, so, we are not really leaking any info by showing the analytics.
However, if running these 3 more queries is not adding any value for a non-group member, I'd rather that we don't show it. What do you think? :)
@aakriti Would you mind attaching a screenshot and highlighting the information you think should not appear? I am still new enough to the product that it helps me to see it.
Thanks for the productive call @jshackelford . It is a lot quicker to go over these issues over a call!
Here is a summary of our discussion:
For now, we will not show the recent activity container to a user who is not a member of a group.
Currently the bug in production would show an error banner when such a user would land on the group's page. The fix for this is ready and under review. (of course the feature flag is not on, so no one can see this error.)
In future we want to consider allowing non-group members to also see these analytics, as a potential entry point for anonymous/non-customers into paid Analytics features. I have started an issue for this
@aakriti.gupta Thanks for keeping this issue updated. FYI - We're also close to getting the New Members created metric added on the FE in !28093 (merged)
I have an MR in review !29165 (merged) so that can make this feature a beta feature. That will allow us to enable the feature per group and eventually, enable it globally.
And there is another follow-up MR !29261 (merged) that we can merge later, to get this feature completely out of beta and remove all feature flag checks. I just have it prepared in advance.
@aakriti.gupta Can we verify that this is working as expected on production? I believe this has already been deployed but I cannot test this as it requires a paid plan.
Update: !29165 (merged) that makes this feature a beta feature is now merged. We are going to release this in beta in 12.10 and in the next release we can move this to GA.
This feature is now enabled for gitlab-com and gitlab-org on staging with the following commands on Slack:
/chatops run feature set group_activity_analytics false --staging/chatops run feature set --group=gitlab-org group_activity_analytics true --staging/chatops run feature set --group=gitlab-com group_activity_analytics true --staging
@jshackelford , here is a little more context for you on the current status and remaining work:
Current status
Right now this feature is available for Starter and above in EE as a beta feature, i.e. disabled by default.
On gitlab.com, it is enabled for 2 groups.
Todo
On gitlab.com we want accounts on Bronze tier and above to have this feature, but right now there is a bug, because of which, when we enable the feature globally on gitlab.com, it gets enabled for all accounts - free tiers as well. This bug is noted in this issue which should be the next top priority work for this epic.
When we're ready to move this feature our of beta, we need to merge in this MR that removes the feature flag entirely. The MR has been code reviewed and only needs a maintainer review and merge.
@bma we're still verifying in gitlab-com and gitlab-org on production since @pshutsin resolved the blocking bugfix issue yesterday. I think once @pshutsin gives us his we can enable for all groups on production.
@fseifoddini it seems like the underlying problem is that "upcoming" page is showing open issues regardless of whether they are in an upcoming milestone or a past milestone.
@pshutsin I bet that page is using a Ruby helper method we could modify. Would you have time to investigate?