Geo: LFS files stopped synching due to duplicate inserts
Zendesk: https://gitlab.zendesk.com/agent/tickets/162494
In geo.log, we see these duplicate errors happened in close proximity:
{
"severity": "ERROR",
"time": "2020-06-29T07:13:19.558Z",
"correlation_id": "83b5e65ae1f07e07880aa5d217747c7f",
"class": "Geo::RegistryConsistencyService",
"host": "***",
"message": "Error while backfilling Geo::LfsObjectRegistry",
"job_id": "c1b0a016861ae61fa1cc4ac4",
"error": "PG::UniqueViolation: ERROR: duplicate key value violates unique constraint \"index_lfs_object_registry_on_lfs_object_id\"\nDETAIL: Key (lfs_object_id)=(175772) already exists.\n"
}
{
"severity": "ERROR",
"time": "2020-06-29T07:13:19.559Z",
"correlation_id": "83b5e65ae1f07e07880aa5d217747c7f",
"class": "Geo::Secondary::RegistryConsistencyWorker",
"host": "***",
"message": "Error while backfilling all",
"job_id": "c1b0a016861ae61fa1cc4ac4",
"error": "PG::UniqueViolation: ERROR: duplicate key value violates unique constraint \"index_lfs_object_registry_on_lfs_object_id\"\nDETAIL: Key (lfs_object_id)=(175772) already exists.\n"
}This suggests a single SQL statement may have contained duplicate keys. In the PostgreSQL logs, you can see that 27151 is included twice:
INSERT INTO "lfs_object_registry" (
"created_at", "retry_at", "bytes",
"lfs_object_id", "retry_count",
"missing_on_primary", "success",
"sha256"
)
VALUES
(
'2020-06-28 21:28:05.082523', NULL,
NULL, 27025, NULL, FALSE, FALSE, NULL
),
(
'2020-06-28 21:28:05.082722', NULL,
NULL, 27026, NULL, FALSE, FALSE, NULL
),
(
'2020-06-28 21:28:05.082810', NULL,
NULL, 27033, NULL, FALSE, FALSE, NULL
),
(
'2020-06-28 21:28:05.082870', NULL,
NULL, 27034, NULL, FALSE, FALSE, NULL
),
(
'2020-06-28 21:28:05.082934', NULL,
NULL, 27035, NULL, FALSE, FALSE, NULL
),
(
'2020-06-28 21:28:05.082999', NULL,
NULL, 27036, NULL, FALSE, FALSE, NULL
),
(
'2020-06-28 21:28:05.083063', NULL,
NULL, 27037, NULL, FALSE, FALSE, NULL
),
(
'2020-06-28 21:28:05.083126', NULL,
NULL, 27038, NULL, FALSE, FALSE, NULL
),
(
'2020-06-28 21:28:05.083192', NULL,
NULL, 27039, NULL, FALSE, FALSE, NULL
),
(
'2020-06-28 21:28:05.083256', NULL,
NULL, 27040, NULL, FALSE, FALSE, NULL
),
(
'2020-06-28 21:28:05.083320', NULL,
NULL, 27046, NULL, FALSE, FALSE, NULL
),
(
'2020-06-28 21:28:05.083386', NULL,
NULL, 27047, NULL, FALSE, FALSE, NULL
),
(
'2020-06-28 21:28:05.083450', NULL,
NULL, 27054, NULL, FALSE, FALSE, NULL
),
(
'2020-06-28 21:28:05.083514', NULL,
NULL, 27059, NULL, FALSE, FALSE, NULL
),
(
'2020-06-28 21:28:05.083583', NULL,
NULL, 27150, NULL, FALSE, FALSE, NULL
),
(
'2020-06-28 21:28:05.083640', NULL,
NULL, 27151, NULL, FALSE, FALSE, NULL
),
(
'2020-06-28 21:28:05.083712', NULL,
NULL, 27151, NULL, FALSE, FALSE, NULL
),
(
'2020-06-28 21:28:05.083782', NULL,
NULL, 27060, NULL, FALSE, FALSE, NULL
),
(
'2020-06-28 21:28:05.083854', NULL,
NULL, 27062, NULL, FALSE, FALSE, NULL
),
(
'2020-06-28 21:28:05.083923', NULL,
NULL, 27077, NULL, FALSE, FALSE, NULL
),
(
'2020-06-28 21:28:05.083984', NULL,
NULL, 27076, NULL, FALSE, FALSE, NULL
),
(
'2020-06-28 21:28:05.084073', NULL,
NULL, 27066, NULL, FALSE, FALSE, NULL
),
(
'2020-06-28 21:28:05.084153', NULL,
NULL, 27080, NULL, FALSE, FALSE, NULL
),
(
'2020-06-28 21:28:05.084224', NULL,
NULL, 27063, NULL, FALSE, FALSE, NULL
),
(
'2020-06-28 21:28:05.084288', NULL,
NULL, 27070, NULL, FALSE, FALSE, NULL
),
(
'2020-06-28 21:28:05.084366', NULL,
NULL, 26993, NULL, FALSE, FALSE, NULL
),
(
'2020-06-28 21:28:05.084432', NULL,
NULL, 27008, NULL, FALSE, FALSE, NULL
),
(
'2020-06-28 21:28:05.084491', NULL,
NULL, 26990, NULL, FALSE, FALSE, NULL
),
(
'2020-06-28 21:28:05.084559', NULL,
NULL, 26879, NULL, FALSE, FALSE, NULL
),
(
'2020-06-28 21:28:05.084617', NULL,
NULL, 26979, NULL, FALSE, FALSE, NULL
),
(
'2020-06-28 21:28:05.084684', NULL,
NULL, 26986, NULL, FALSE, FALSE, NULL
),
(
'2020-06-28 21:28:05.084748', NULL,
NULL, 26988, NULL, FALSE, FALSE, NULL
),
(
'2020-06-28 21:28:05.084814', NULL,
NULL, 26991, NULL, FALSE, FALSE, NULL
),
(
'2020-06-28 21:28:05.084877', NULL,
NULL, 27006, NULL, FALSE, FALSE, NULL
),
(
'2020-06-28 21:28:05.084939', NULL,
NULL, 26980, NULL, FALSE, FALSE, NULL
),
(
'2020-06-28 21:28:05.085008', NULL,
NULL, 27007, NULL, FALSE, FALSE, NULL
),
(
'2020-06-28 21:28:05.085065', NULL,
NULL, 27009, NULL, FALSE, FALSE, NULL
),
(
'2020-06-28 21:28:05.085128', NULL,
NULL, 27026, NULL, FALSE, FALSE, NULL
),
(
'2020-06-28 21:28:05.085195', NULL,
NULL, 27025, NULL, FALSE, FALSE, NULL
),
(
'2020-06-28 21:28:05.085257', NULL,
NULL, 27025, NULL, FALSE, FALSE, NULL
),
(
'2020-06-28 21:28:05.085328', NULL,
NULL, 27026, NULL, FALSE, FALSE, NULL
),
(
'2020-06-28 21:28:05.085386', NULL,
NULL, 27046, NULL, FALSE, FALSE, NULL
),
(
'2020-06-28 21:28:05.085453', NULL,
NULL, 27047, NULL, FALSE, FALSE, NULL
),
(
'2020-06-28 21:28:05.085526', NULL,
NULL, 27025, NULL, FALSE, FALSE, NULL
),
(
'2020-06-28 21:28:05.085589', NULL,
NULL, 27026, NULL, FALSE, FALSE, NULL
),
(
'2020-06-28 21:28:05.085657', NULL,
NULL, 26871, NULL, FALSE, FALSE, NULL
),
(
'2020-06-28 21:28:05.085714', NULL,
NULL, 26869, NULL, FALSE, FALSE, NULL
),
(
'2020-06-28 21:28:05.085778', NULL,
NULL, 26863, NULL, FALSE, FALSE, NULL
),
(
'2020-06-28 21:28:05.085841', NULL,
NULL, 26866, NULL, FALSE, FALSE, NULL
),
(
'2020-06-28 21:28:05.085914', NULL,
NULL, 26867, NULL, FALSE, FALSE, NULL
),
(
'2020-06-28 21:28:05.085977', NULL,
NULL, 26868, NULL, FALSE, FALSE, NULL
),
(
'2020-06-28 21:28:05.086041', NULL,
NULL, 26870, NULL, FALSE, FALSE, NULL
) RETURNING "id"This suggests find_registry_differences may be including dupes? https://gitlab.com/gitlab-org/gitlab/blob/d27df23488648bd313621661d22a9d180457f2dc/ee/app/services/geo/registry_consistency_service.rb#L55-61
Edited by Catalin Irimie