Allow Product Analytics data to be exported to JSON

Release notes

Problem to solve

More context in parent epic.

Proposal

Create a way for customers to be able to retrieve the product analytics event data as a JSON file.

  • This is intended to be used primarily for making backups for their data or for exporting it to move to another tool, rather than a common workflow in day-to-day usage.
  • This is intended to be a first, boring solution to provide export capabilities.

Consider a few different ways to approach this (the below are ideas for consideration during planning breakdown, not requirements):

  • Create a service that exposes a single API which causes it to query Cube.dev for all user data, package it, and then send an email with where it can be downloaded. This will be good in case the export is large and takes a long time to generate. Retain it for a short period of time then delete it.
  • Create a proxy server to sit in front of Cube to only allow access to the export functionality.
  • Other ideas

Requirements

  1. Provide an API that can be called to retrieve the stored events as a JSON file
    • Consider that this may be a large data set when exporting, so design the API for scalability. Consider things like pagination as a way to help with this.
    • This API should connect to the currently configured Product Analytics cluster. If one is not available, return an error.
    • Users should have to have Maintainer access or higher to use this API. Return an error if not.
  2. Provide a GUI-based way to retrieve a JSON file containing the events.
    • Consider leveraging the API built in the previous requirement for this piece.
    • Note: Intention is to keep this simple. This could be an Export button under project settings that displays a modal with an email address for where to send the results. If this turns out to be complicated, defer the GUI approach to a future issue and start with an API-only approach to start.
  3. Direct exposure of ClickHouse and Cube.dev endpoints is not how this should be approached. We do not want to give direct access to those systems.
  4. Add Service Ping instrumentation to record usage of this API.

Intended users

Feature Usage Metrics

This page may contain information related to upcoming products, features and functionality. It is important to note that the information presented is for informational purposes only, so please do not rely on the information for purchasing or planning purposes. Just like with all projects, the items mentioned on the page are subject to change or delay, and the development, release, and timing of any products, features, or functionality remain at the sole discretion of GitLab Inc.

Edited by Sam Kerr