Create new configurator endpoint and migration for funnels
Problem to solve
Right now, the funnels Cube logic tries to retrieve the user-generated funnels definitions from the GitLab API. The problem with this is that we require a GitLab API token to be given to Cube for it to retrieve these definitions. Since this token is only available when a user sends a request, not when Cube makes the call in the background, we're getting inconsistency with the API response. This results in Cube not always being aware of the funnel definitions for a project.
Proposed solution
We should create a new configurator endpoint which accepts funnel definitions submitted from .com and stores these definitions in the Clickhouse database. Cube will then retrieve these definitions from Clickhouse directly. See #427860 (comment 1688776812)+ for more background details.
The ability to remove funnel definitions will be a future iteration: Update funnels configurator endpoint to remove ... (#438038 - closed).
Restricting access to this endpoint will be a future iteration: Restrict access to analytics configurator endpo... (#438034 - closed).
Table structure
The proposed table structure would look something like:
funnel_schemas
Field name | Field type | Required |
---|---|---|
name |
String |
true |
contents |
string |
true |
created_at |
DateTime64(3, 'GMT') |
true |
updated_at |
DateTime64(3, 'GMT') |
true |
With the primary key of PRIMARY KEY name
.
CREATE TABLE funnel_schemas
(
name String,
contents String,
created_at DateTime64(3, 'GMT'),
updated_at DateTime64(3, 'GMT')
)
ENGINE = MergeTree
PRIMARY KEY name;
Having the created_at
and updated_at
gives us information for debugging and auditing purposes. I've opted for using the name
as the primary key, but we could also require the user to define an ID instead.
Endpoint requirements
The endpoint should accept a JSON payload which contains:
- The project ID(s) which these funnels should be stored against (so we can get the correct project DB).
- Any array of funnels which contains:
- The funnel definition:
- Funnel name
- Schema contents
- The state of the funnel:
- Created
- Updated
- Deleted
- If the funnel was updated, what the old funnel name was.
- The funnel definition:
- The endpoint should then apply the correct
INSERT
,ALTER
,DELETE
SQL functions based upon the state of the funnel.
3️⃣
Implementation plan - - For new projects, add a new set-up method for the proposed table structure similar to
handleSetupProject.go
. - For existing projects, add a new migration for the proposed table structure similar to
20231018112205_migrate_existing_projects_with_sessions.go
. - Add a new route to
routes.go
. - Add a new handler to
/handlers
similar tohandleProjectUsage.go
.- The handler should accept a JSON payload that meets the endpoint requirements.
- In the event that the handler receives an array of project IDs, it should apply the funnels to each project's DB.
- The handler should save the data accordingly.
- The handler should return a
200 OK
when successful. - The handler should return a
400 Bad Request
or500 Internal Server Error
JSON response depending upon the error.
- The handler should accept a JSON payload that meets the endpoint requirements.
- Add tests to
/tests
.