Add interface to create analytics stack for project (!95999) · Merge requests · GitLab.org / GitLab

Max Woolf requested to merge mw/gl-project-to-jitsu-poc into master Aug 23, 2022

What does this MR do and why?

Adds an initialize_analytics_stack method to EE::Project that sets up the connection between a GitLab project, Jitsu, Clickhouse and Cube as defined in the groupproduct analytics issue: Discussion: Project -> Jitsu Mapping (gitlab-org/analytics-section/product-analytics/general-discussion#2 - closed)
Adds a bunch of application settings required to perform those actions, including a password for jitsu which is stored as an encrypted variable.
Pushes the process itself in to a background worker.

Database Review

Migrate

main: == 20220818125332 AddJitsuTrackingColumnsToApplicationSettings: migrating =====
main: -- add_column(:application_settings, :jitsu_host, :text)
main:    -> 0.0028s
main: -- add_column(:application_settings, :jitsu_project_xid, :text)
main:    -> 0.0005s
main: -- add_column(:application_settings, :clickhouse_connection_string, :text)
main:    -> 0.0004s
main: -- add_column(:application_settings, :jitsu_administrator_email, :text)
main:    -> 0.0004s
main: -- add_column(:application_settings, :encrypted_jitsu_administrator_password, :binary)
main:    -> 0.0004s
main: -- add_column(:application_settings, :encrypted_jitsu_administrator_password_iv, :binary)
main:    -> 0.0004s
main: == 20220818125332 AddJitsuTrackingColumnsToApplicationSettings: migrated (0.0057s)

main: == 20220818125703 AddJitsuTrackingColumnsToApplicationSettingsTextLimits: migrating
main: -- transaction_open?()
main:    -> 0.0000s
main: -- current_schema()
main:    -> 0.0011s
main: -- transaction_open?()
main:    -> 0.0000s
main: -- execute("ALTER TABLE application_settings\nADD CONSTRAINT check_dea8792229\nCHECK ( char_length(jitsu_host) <= 255 )\nNOT VALID;\n")
main:    -> 0.0010s
main: -- current_schema()
main:    -> 0.0002s
main: -- execute("SET statement_timeout TO 0")
main:    -> 0.0001s
main: -- execute("ALTER TABLE application_settings VALIDATE CONSTRAINT check_dea8792229;")
main:    -> 0.0006s
main: -- execute("RESET statement_timeout")
main:    -> 0.0002s
main: -- transaction_open?()
main:    -> 0.0000s
main: -- current_schema()
main:    -> 0.0001s
main: -- transaction_open?()
main:    -> 0.0000s
main: -- execute("ALTER TABLE application_settings\nADD CONSTRAINT check_fc732c181e\nCHECK ( char_length(jitsu_project_xid) <= 255 )\nNOT VALID;\n")
main:    -> 0.0005s
main: -- current_schema()
main:    -> 0.0001s
main: -- execute("ALTER TABLE application_settings VALIDATE CONSTRAINT check_fc732c181e;")
main:    -> 0.0005s
main: -- transaction_open?()
main:    -> 0.0000s
main: -- current_schema()
main:    -> 0.0001s
main: -- transaction_open?()
main:    -> 0.0000s
main: -- execute("ALTER TABLE application_settings\nADD CONSTRAINT check_d4865d70f3\nCHECK ( char_length(clickhouse_connection_string) <= 255 )\nNOT VALID;\n")
main:    -> 0.0005s
main: -- current_schema()
main:    -> 0.0001s
main: -- execute("ALTER TABLE application_settings VALIDATE CONSTRAINT check_d4865d70f3;")
main:    -> 0.0005s
main: -- transaction_open?()
main:    -> 0.0000s
main: -- current_schema()
main:    -> 0.0001s
main: -- transaction_open?()
main:    -> 0.0000s
main: -- execute("ALTER TABLE application_settings\nADD CONSTRAINT check_ec3ca9aa8d\nCHECK ( char_length(jitsu_administrator_email) <= 255 )\nNOT VALID;\n")
main:    -> 0.0005s
main: -- current_schema()
main:    -> 0.0001s
main: -- execute("ALTER TABLE application_settings VALIDATE CONSTRAINT check_ec3ca9aa8d;")
main:    -> 0.0005s
main: == 20220818125703 AddJitsuTrackingColumnsToApplicationSettingsTextLimits: migrated (0.0220s)

Rollback

main: == 20220818125703 AddJitsuTrackingColumnsToApplicationSettingsTextLimits: reverting
main: -- transaction_open?()
main:    -> 0.0000s
main: -- transaction_open?()
main:    -> 0.0000s
main: -- execute("ALTER TABLE application_settings\nDROP CONSTRAINT IF EXISTS check_dea8792229\n")
main:    -> 0.0010s
main: -- transaction_open?()
main:    -> 0.0000s
main: -- transaction_open?()
main:    -> 0.0000s
main: -- execute("ALTER TABLE application_settings\nDROP CONSTRAINT IF EXISTS check_fc732c181e\n")
main:    -> 0.0006s
main: -- transaction_open?()
main:    -> 0.0000s
main: -- transaction_open?()
main:    -> 0.0000s
main: -- execute("ALTER TABLE application_settings\nDROP CONSTRAINT IF EXISTS check_d4865d70f3\n")
main:    -> 0.0005s
main: -- transaction_open?()
main:    -> 0.0000s
main: -- transaction_open?()
main:    -> 0.0000s
main: -- execute("ALTER TABLE application_settings\nDROP CONSTRAINT IF EXISTS check_ec3ca9aa8d\n")
main:    -> 0.0005s
main: == 20220818125703 AddJitsuTrackingColumnsToApplicationSettingsTextLimits: reverted (0.0136s)

main: == 20220818125332 AddJitsuTrackingColumnsToApplicationSettings: reverting =====
main: -- remove_column(:application_settings, :encrypted_jitsu_administrator_password_iv, :binary)
main:    -> 0.0021s
main: -- remove_column(:application_settings, :encrypted_jitsu_administrator_password, :binary)
main:    -> 0.0005s
main: -- remove_column(:application_settings, :jitsu_administrator_email, :text)
main:    -> 0.0003s
main: -- remove_column(:application_settings, :clickhouse_connection_string, :text)
main:    -> 0.0003s
main: -- remove_column(:application_settings, :jitsu_project_xid, :text)
main:    -> 0.0003s
main: -- remove_column(:application_settings, :jitsu_host, :text)
main:    -> 0.0003s
main: == 20220818125332 AddJitsuTrackingColumnsToApplicationSettings: reverted (0.0057s)

Screenshots or screen recordings

Screenshots are required for UI changes, and strongly recommended for all other merge requests.

How to set up and validate locally

Set up the devkit and start it running. docker-compose up
In this branch, run rails c to open the rails console

Enable the feature flag

Feature.enable(:jitsu_connection_proof_of_concept)

Set the required application settings:

The Jitsu attributes are set up when first configuring your jitsu instance.

ApplicationSetting.current.update(jitsu_project_xid: 'XXXXXXXXXX', jitsu_host: 'http://localhost:8000', clickhouse_connection_string: 'http://test:test@clickhouse:8123', jitsu_administrator_email: 'XXXXXXX', jitsu_administrator_password: 'XXXXXXX')
reload!

Find a project on the console: a = Project.find(PROJECT_ID)
Run ProductAnalytics::InitializeStackService.new(container: a).execute
Ensure that a sidekiq job was enqueued.
In the Jitsu configurator, ensure that a new API key (with the project ID) has been created, along with a new associated connection to a clickhouse destination. 👇

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

I have evaluated the MR acceptance checklist for this MR.

Edited Sep 14, 2022 by Max Woolf

Add interface to create analytics stack for project