Fix ActiveRecord::IrreversibleOrderError during restore from backup
Due to the frequency with which settings are accessed, it is likely that
during a backup restore a running GitLab process will insert a new
application_settings
row before the constraints have been added to the
table. This would add an extra row with ID 1 and prevent the primary key
constraint from being added, which made ActiveRecord
throw a
IrreversibleOrderError
anytime the settings were accessed.
The following sequence was seen in PostgreSQL logs:
- Restore:
CREATE TABLE public.application_settings ...
- Restore:
CREATE SEQUENCE public.application_settings_id_seq
- Puma/Sidekiq:
INSERT INTO "application_settings" ...
- Restore:
COPY public.application_settings (id, ...
- Restore:
ADD CONSTRAINT application_settings_pkey PRIMARY KEY (id);
In step 3, since GitLab inserted a new row with ID 1, but shortly after
that the restore process would also copy another row with ID 1. As a
result, the ADD CONSTRAINT
in step 5 would fail with the message:
ERROR: could not create unique index "application_settings_pkey"
Key (id)=(1) is duplicated.
The right way to fix this would be to restore the database in a single
transaction via the --single-transaction
flag in psql
, but this
usually fails due to permission errors when extensions and schemas are
dropped and recreated. psql
generally works best with superuser
access, but restore usually runs with limited access permissions.
GitLab normally inserts an application_settings
row if no row is
found, but to prevent duplicate rows from being inserted we ensure that
a primary key has been set beforehand. This should help prevent primary
key collisions and duplicate rows from being added.
We still need to tell administrators to stop all processes that connect to the database (puma, sidekiq, and gitlab-exporter), but this check should ensure that a primary key constraint is added during a restore.
Relates to #36405 (closed)