Skip to content

Fix ActiveRecord::IrreversibleOrderError during restore from backup

Stan Hu requested to merge sh-fix-backup-restore-race into master

Due to the frequency with which settings are accessed, it is likely that during a backup restore a running GitLab process will insert a new application_settings row before the constraints have been added to the table. This would add an extra row with ID 1 and prevent the primary key constraint from being added, which made ActiveRecord throw a IrreversibleOrderError anytime the settings were accessed.

The following sequence was seen in PostgreSQL logs:

  1. Restore: CREATE TABLE public.application_settings ...
  2. Restore: CREATE SEQUENCE public.application_settings_id_seq
  3. Puma/Sidekiq: INSERT INTO "application_settings" ...
  4. Restore: COPY public.application_settings (id, ...
  5. Restore: ADD CONSTRAINT application_settings_pkey PRIMARY KEY (id);

In step 3, since GitLab inserted a new row with ID 1, but shortly after that the restore process would also copy another row with ID 1. As a result, the ADD CONSTRAINT in step 5 would fail with the message:

ERROR:   could not create unique index "application_settings_pkey"
Key (id)=(1) is duplicated.

The right way to fix this would be to restore the database in a single transaction via the --single-transaction flag in psql, but this usually fails due to permission errors when extensions and schemas are dropped and recreated. psql generally works best with superuser access, but restore usually runs with limited access permissions.

GitLab normally inserts an application_settings row if no row is found, but to prevent duplicate rows from being inserted we ensure that a primary key has been set beforehand. This should help prevent primary key collisions and duplicate rows from being added.

We still need to tell administrators to stop all processes that connect to the database (puma, sidekiq, and gitlab-exporter), but this check should ensure that a primary key constraint is added during a restore.

Relates to #36405 (closed)

Edited by Stan Hu

Merge request reports