Phase 2 - Replace old events and contributions tables

#95 (closed) introduced rebuilt_events and rebuilt_contributions tables. We should use the new tables whenever this data is needed on Rails app.

  1. Add a feature flag on all places where fetch events and contributions tables to get data from rebuilt tables, keep in mind we need to take organization_id into account when using the new tables.

We added traversal_path dictionaries lookup to Siphon tables. Additional steps are needed to complete this issue. The events_new table is no longer needed.

Before Siphon is in production

  1. Drop events_new table and events_new_mv materialized view
  2. Drop siphon_events table
  3. Recreate siphon_events table with dictionary lookup
rails generate gitlab:click_house:siphon events --with-traversal-path
  1. Recreate contributions_new_mv to read data from the new siphon_events table.
-- Drop the existing materialized view
DROP VIEW IF EXISTS gitlab_clickhouse_development.contributions_new_mv;

-- Recreate with siphon_events as source
CREATE MATERIALIZED VIEW gitlab_clickhouse_development.contributions_new_mv 
TO gitlab_clickhouse_development.contributions_new
(
    `id` Int64,
    `path` String,
    `author_id` UInt64,
    `target_type` String,
    `action` UInt8,
    `created_at` Date,
    `updated_at` Date,
    `deleted` Bool,
    `version` DateTime64(6, 'UTC')
)
AS SELECT
    id,
    argMax(path, siphon_events.version) AS path,
    argMax(author_id, siphon_events.version) AS author_id,
    argMax(target_type, siphon_events.version) AS target_type,
    argMax(action, siphon_events.version) AS action,
    argMax(DATE(created_at), siphon_events.version) AS created_at,
    argMax(DATE(updated_at), siphon_events.version) AS updated_at,
    argMax(deleted, siphon_events.version) AS deleted,
    max(siphon_events.version) AS version
FROM gitlab_clickhouse_development.siphon_events
WHERE ((siphon_events.action IN (5, 6)) AND (siphon_events.target_type = '')) 
   OR ((siphon_events.action IN (1, 3, 7, 12)) AND (siphon_events.target_type IN ('MergeRequest', 'Issue', 'WorkItem')))
GROUP BY id

After Siphon is in production

  1. Backfill contributions_new table with data (this will probably be done with Siphon initial snapshot).
  2. Rollout the feature flag [Feature flag] Rollout of fetch_contributions_d... (gitlab-org/gitlab#601099)
  3. Replace current contributions table with contributions_new
  4. Remove ClickHouse::EventsSyncWorker
  5. Update contribution analytics data flow documentation.
Edited by Felipe Cardozo