Discussion: Project -> Jitsu Mapping
Overview
Discussion around the best way to map jitsu API keys to individual projects.
Problem
In the initial PoC, we set up Jitsu with a single API key to ingest all data across any project. In the initial iteration for GitLab.com, we'll be allowing users to track analytics for any project rather than just one test project that we had set up.
Questions to answer
One API key per project?
- Currently it seems sensible to create a new API key for each project that is analyzable within GitLab. This can be done programattaically via background jobs using the Jitsu configurator API. See example: #2 (comment 1069416425)
- There are outstanding questions about the most reliable way to do this which may be answered by the next point.
Clickhouse table/databases per project?
How should we segment the clickhouse setup for each newly tracked project? Perhaps for development of the SaaS solution, we should segment also by datbase table so instead of the M-1 relationship we have between API keys and clickhouse destinations here, we should have a 1-1 relationship between API keys and clickhouse destinations where we use the same clickhouse database, but a different table within it.
Proposed technical solution/workflow
graph TD
A[Project Owner] --> |Decides to add Analytics to | B(Project)
B --> D[Generate Jitsu auth token]
D --> |used to create| E[Jitsu API key]
E --> |Persist key to| B
D --> |used to create| F[Clickhouse destination - table per project]
E --> |associate with| F
- User creates GitLab project
- Enables product analytics
- Background job generates a jitsu configurator access token
- Access token is used to create a new jitsu API key.
- Access token is used to create a new clickhouse destination. (1-1 relationship between API key and destination)
- Note: We're still planning to use a single clickhouse deployment, but each jitsu destination should refer to a unique clickhouse table for easier data segmenting.
- Access token is used to associate API key with destination. (This effectively removes the issue noted in point 3 here)
Edited by Sam Kerr