Skip to content

Convert events.id to bigint - Step 1: Add new columns and sync data

events is the largest table in GitLab's database that is defined using an integer (int4) Primary Key.

On November 24th, we are at 47% capacity used

# select current_date, max(id) as current_max, 2147483647 as max_integer, ((max(id) / 2147483647.0) * 100)::integer as percentage_used from events;
 current_date | current_max | max_integer | percentage_used
--------------+-------------+-------------+-----------------
 2020-11-24   |  1003520786 |  2147483647 |              47

The first step to address the problem is to create a new column of type bigint, load all the data by using background jobs from the current id column to it and keep them in sync with a trigger.

We have to also follow the same approach for all tables that reference events.id, in this case push_event_payloads.event_id

So the overall process in %13.7 will be as follows:

  • Create new column events.new_id, with bigint NOT NULL DEFAULT 0

  • Install sync triggers (install_rename_triggers should work here)

  • Start background CopyColumn jobs (we'll update it to use our new background migration jobs framework if we have not)

  • Same for the column push_event_payloads.event_id that references events.id

We'll follow that with a cleanup migration in %13.7 and then add the necessary indexes and swap the PK and FK.

Related Tracking Issue: #220023 (closed)

Edited by Yannis Roussos