Spike Research: Explore possible POC for VSA Jira-based stages

Overview

groupoptimize vision is to make GitLab the tool of choice for data-driven DevOps organizations. To achieve this one of our goals is to make Value Stream Analytics (VSA) the single source of truth (SSOT) for DevOps Analytics and to allow users to aggregate multiple data records into one VSA.

Problem to solve

Customers that are using Jira for issue management and GitLab for CI\CD need single source of insight about what's happening within the entire software delivery lifecycle.

Assumption

  1. GitLab SaaS first with Atlassian cloud.
  2. PostgreSQL is a blocker for this integration, and a new datastore is needed - Draft: Spike Research: ClickHouse as the Datast... (#367178 - closed)
  3. Authentication and authorization will be done with API token.
  4. No support for "mixed" stages - start and stop events on API-based stages can be from the API.
  5. No bidirectional integration - data flow is Jira--> GL only. VSA will only visualize the Jira data.
  6. Data refresh will be in near real-time - updates every 15m.
  7. VSA-Jira integration needs to be compatible with other GitLab-Jira integrations.
  8. Group-level only.
  9. About VSA DORA - we will start with only "Deployment Frequency" and "Lead Time for Changes".
  10. "Tasks by Type" chart is out of scope.

Reference use cases

  1. Basic flow for custom VSA stages base on Jira events. :

    • Task\issue create in JIRA
    • Branch created in GL
    • Merge request is merged in GL
    • Task\issue closed in JIRA

  2. Common flow - TBD

  3. Advanced flow - TBD

Research Goals

  1. Outline a possible POC architecture, major steps, and effort estimation for POC the basic flow.
  2. Summarize the main gaps in delivering this integration for customers.

Investigation and clarification questions:

  1. Is it possible to use some of the existing Jira integration technical design?
  • Can we mapped the events using the Import Jira tool?
  • In the Jira-based stages can we have a link to view Jira issues for deep-dive stage performance investigation?
  1. Any other tools from Ecosystem:Integrations](https://about.gitlab.com/direction/ecosystem/integrations/) group that can be relevant to solving this problem?
  2. Can we add filtering to the API? group, project, label, milestone?
  3. What enhancements need to be done for the VSA stage event schema?
  4. For Jira-based stages can we display in the Work items table the Jira issue name, create time, last event and duration?
  5. Is it possible to calculate all the metrics in the VSA Overview dashboard? - Lead Time, Cycle Time, New Issues and Deploys?

Investigation summary

Proposed draft workflow:

Before we start: client must have Maintainer+ access level API token.

  1. Client creates custom "external" stage with unique start & end event identifiers (e.g. "issue_created" and "issue_closed")
  2. When stage end event happens customer sends following mutation request to GitLab GraphQL:
addExternalValueStreamAnalyticsEvent(input: inputParams)

with following inputParams

param Type Required? Description
groupPath ID ✅ full path of group where you want to see this VSA event
externalId String ✅ string containing global unique ID of Jira issue
startEventId String ✅ String from stage configuration. (e.g. "issue_created")
endEventId String ✅ String from stage configuration. (e.g. "issue_closed")
startEventTimestamp Timestamp ✅ Timestamp with timezone when start event happened
endEventTimestamp Timestamp ✅ Timestamp with timezone when end event happened
labels Array of strings Issue labels for filtering. Missing labels will be created at GitLab if not found
authorUsername String Issue author username. Group member with such username must exist at GitLab
assigneeUsernames Array of Strings Issue assignees usernames. Group members with such username must exist at GitLab
milestone String ID of associated milestone. Milestone must exist at GitLab
externalUrl String ? String containing URL to Jira issue

If response is successful then everything is OK and VSA will be refreshed soon to reflect new data.

  • More details can be found here

Assumptions, Impact and gaps can be found

  1. Customers write their custom stage events to GitLab and filter & retrieve it later on. It's on customers shoulders to decide WHEN exactly particular event happened outside of GitLab and safely deliver it to GitLab via API with all required data.
  2. Users syncing between Jira and GitLab is mandatory requirement for this integration.
  3. Tasks by Type chart - This chart won't work. The chart requires real native issues to be created and doesn't rely on _events table at all.
Edited by Haim Snir