chore(data-insights-platform): Update deployment timeline

creation-date: "2025-04-10"

authors: [ "@ankitbhatnagar" ]

coach: [ "@ahegyi" ]

approvers: [ "@ghosh-abhinaba" ]

approvers: [ "@chopras" ]

owning-stage: "~section::analytics"

participating-stages: []

toc_hide: true

Following is a brief description of each underlying component. Note, we have also compiled _architecture blueprints_ for some of these individually to ensure we can deep-dive into further details of each component and gather valuable feedback from a wider audience.

- __Ingester(s)__: Single ingestion mechanism for supported event types - which can be run both locally for development & as a cluster when in production. This layer is intentionally stateless to allow horizontal scalability to allow ingesting large data volumes. We aim to build the support for ingesting the following event-types:

  - [Snowplow](https://docs.snowplow.io/docs/fundamentals/events/) _(current iteration)_

  - [CloudEvents](https://cloudevents.io/) _(next iteration)_

  - [Snowplow](https://docs.snowplow.io/docs/fundamentals/events/)

  - [CloudEvents](https://cloudevents.io/)

  - [Service Ping](https://docs.gitlab.com/development/internal_analytics/service_ping/) _(future iteration)_

- __Siphon__: Custom, in-house Change Data Capture (CDC) implementation to logically replicate data from Postgres into other systems such as ClickHouse.

- __Exporters__: Custom implementations that help ship ingested data into designated persistent stores for further querying/processing.

  - [ClickHouse Exporter](https://gitlab.com/gitlab-org/analytics-section/platform-insights/core/-/tree/main/pkg/snowplow/exporter/clickhouse?ref_type=heads): [ClickHouse](https://clickhouse.com/docs/intro) is our designated persistent database which helps us persist all analytical data ingested by the Platform and query from using the `Query API`.

  - [S3/GCS Exporter](https://gitlab.com/gitlab-org/analytics-section/platform-insights/core/-/issues/19): Having data shipped to `S3`/`GCS` helps land data into Snowflake powering our current analytical query-workflows using Snowflake & Tableau.

  - [Billing Exporter](https://gitlab.com/gitlab-org/analytics-section/platform-insights/core/-/blob/main/pkg/snowplow/exporter/billing/controller.go?ref_type=heads): Helps persist ingested usage billing events into Fulfilment's ClickHouse.

- [__ClickHouse__](https://clickhouse.com/docs/intro): External persistent database that allows for durable persistence and [advanced OLAP querying capabilities](https://clickhouse.com/docs/concepts/olap) for all analytical data ingested within the Platform.

- __Query API__: Custom, semantic querying layer for clients/UIs to query data persisted within ClickHouse.

- __Query API__: Custom, semantic querying layer for clients/UIs to query data persisted within Data Insights Platform instances.

  - [Architecture Blueprint introducing Platform's querying capabilities](/handbook/engineering/architecture/design-documents/data_insights_platform_querying_api/).

### Identified use-cases

| Dedicated | One dedicated cluster per instance |

| Self-Managed | Standalone cluster subject to distribution/instance-sizing |

## Rollout roadmap

### Deployments Timeline

> Last Update: 2026 Q3

| Deliverables | Timeline | Description |

|---|---|---|

| Pre-staging/Testing | FY26Q1 | Deployed to GKE using [this project](https://gitlab.com/gitlab-org/analytics-section/platform-insights/data-insights-platform-sandbox). |

| `.com` Staging | TBD | - |

| `.com` Production | TBD | - |

| Gradual rollout to Dedicated | TBD | - |

| Gradual rollout to Self-managed | TBD | - |

| First test instance on `pre-staging` | 2025 Q1 | Deployed to GKE using [this project](https://gitlab.com/gitlab-org/analytics-section/platform-insights/data-insights-platform-sandbox) and benchmarked successfully. |

| DIP-powered Usage Billing Infra | 2025 Q3,Q4 | Usage Billing for DAP GA'ed in 2026 Q1, Usage Billing for Dedicated Hosted Runners GA'ed in 2026 Q3. |

| Postgres CDC via Siphon on `.com` | 2026 Q1,Q2 | All requested tables replicated to ClickHouse and made available to GitLab's in-product use-cases. |

| Postgres CDC via Siphon on Dedicated | 2026 Q3 | Ongoing effort as we rollout DIP + Orbit to each Dedicated tenant gradually. |

| Postgres CDC via Siphon on Self-Managed | 2026 Q4 | First SM DIP + Orbit instance deployed successfully using OAK, further expansion planned. |

| Query API for `.com` | 2026 Q4 | Ongoing work to enable querying data landed in ClickHouse via Siphon. |

| DIP-powered Snowplow Data Ingestion | 2026 Q4 | Ongoing work to replace existing Snowplow infrastructure with DIP to ingest & process internal events. |

| Third-party Data Ingestion | 2027 Q1 | Pending R&D to enable ingesting third-party data into GitLab's datalake. |

| External Data Sharing | 2027 Q1 | Pending R&D to enable sharing data from GitLab's datalake to users. |

### Opportunities looking forward

**Current blockers for Golden Path:**

- **Data storage:** Most custom data cannot yet be stored in the product (depends on [Data Insights Platform](/handbook/engineering/architecture/design-documents/data_insights_platform/) - no production estimate yet in [roadmap](/handbook/engineering/architecture/design-documents/data_insights_platform/#rollout-roadmap))

- **Data storage:** Most custom data cannot yet be stored in the product (depends on [Data Insights Platform](/handbook/engineering/architecture/design-documents/data_insights_platform/) - no production estimate yet in [roadmap](/handbook/engineering/architecture/design-documents/data_insights_platform/#deployments-timeline))

- **Custom dashboards:** Not yet available for product data (depends on Data Exploration feature in [Dashboard Foundations epic](https://gitlab.com/groups/gitlab-org/-/epics/18072) - planned for **FY26 Q4**)

- **Alerting:** Real-time alerting on product data not yet planned (though feature teams are open to the idea)