Fix PG::UntranslatableCharacter errors in MoveCiBuildsMetadata background migration
What does this MR do and why?
This merge request addresses PG::UntranslatableCharacter errors in the MoveCiBuildsMetadata background migration by:
- Adding a
SanitizedYamlCoderclass to safely handle YAML deserialization with error recovery - Configuring the
configattribute withJsonPgSafetype for safe JSON handling - Using the custom coder for
optionsandyaml_variablesserialization to prevent encoding issues - Updating the checksum computation to use
JsonPgSafeserialization
These changes ensure that invalid or untranslatable characters in YAML/JSON data are handled gracefully without causing migration failures.
gitlabhq_dblab=# SELECT migration.id, jobs.id, jobs.attempts,jobs.status, transition_logs.exception_class, transition_logs.exception_message FROM batched_background_migrations as migration INNER JOIN batched_background_migration_jobs as jobs ON jobs.batched_background_migration_id = migration.id INNER JOIN batched_background_migration_job_transition_logs as transition_logs ON transition_logs.batched_background_migration_job_id = jobs.id WHERE transition_logs.next_status = '2' AND migration.job_class_name = 'MoveCiBuildsMetadata' AND exception_message <> 'Sidekiq::Shutdown' and migration.id in (3000518, 3000519, 3000520);
id | id | attempts | status | exception_class | exception_message
---------+---------+----------+--------+--------------------------------+-----------------------------------------------------------------------------------------------
3000518 | 3443452 | 0 | 0 | ActiveRecord::StatementInvalid | PG::UntranslatableCharacter: ERROR: unsupported Unicode escape sequence +
| | | | | LINE 1: ...33533643137616337323663633339363435336138646162', '{"options... +
| | | | | ^ +
| | | | | DETAIL: \u0000 cannot be converted to text. +
| | | | | CONTEXT: JSON data, line 1: ...i","public":true},{"key":"YELLOW","value":"\u0000... +
| | | | |
3000518 | 3452575 | 0 | 0 | ActiveRecord::StatementInvalid | PG::UntranslatableCharacter: ERROR: unsupported Unicode escape sequence +
| | | | | LINE 1: ...03339633161623461363165623865383761333864356464', '{"options... +
| | | | | ^ +
| | | | | DETAIL: \u0000 cannot be converted to text. +
| | | | | CONTEXT: JSON data, line 1: ...lic":true},{"key":"DEFAULT_COLOR","value":"\u0000... +
| | | | |
3000518 | 3462457 | 0 | 0 | Psych::SyntaxError | (<unknown>): did not find expected ',' or '}' while parsing a flow mapping at line 1 column 1
3000518 | 3462676 | 0 | 0 | Psych::SyntaxError | (<unknown>): did not find expected ',' or '}' while parsing a flow mapping at line 1 column 1
3000518 | 3462570 | 0 | 0 | Psych::SyntaxError | (<unknown>): did not find expected ',' or '}' while parsing a flow mapping at line 1 column 1
3000518 | 3462679 | 0 | 0 | Psych::SyntaxError | (<unknown>): did not find expected ',' or '}' while parsing a flow mapping at line 1 column 1
3000518 | 3462573 | 0 | 0 | Psych::SyntaxError | (<unknown>): did not find expected ',' or '}' while parsing a flow mapping at line 1 column 1
3000518 | 3462510 | 0 | 0 | Psych::SyntaxError | (<unknown>): did not find expected ',' or '}' while parsing a flow mapping at line 1 column 1Connect to a database clone: GITLAB_SIMULATE_SAAS=1 DISABLE_POSTGRES_PARTITION_CREATION_ON_STARTUP=1 pgai use -o ci -- bin/rails c
Retry the batch jobs:
- example for UntranslatableCharacter:
Gitlab::Database::SharedModel.using_connection(Ci::ApplicationRecord.connection) { Gitlab::Database::BackgroundMigration::BatchedMigrationWrapper.new(connection: Ci::ApplicationRecord.connection).perform(Gitlab::Database::BackgroundMigration::BatchedJob.find(3443452)) } - example for Psych::SyntaxError:
Gitlab::Database::SharedModel.using_connection(Ci::ApplicationRecord.connection) { Gitlab::Database::BackgroundMigration::BatchedMigrationWrapper.new(connection: Ci::ApplicationRecord.connection).perform(Gitlab::Database::BackgroundMigration::BatchedJob.find(3462457)) }
They should start inserting job definition configs.
References
Related to #586270 (closed)
Screenshots or screen recordings
| Before | After |
|---|---|
How to set up and validate locally
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.
Edited by Marius Bobin