Restore AI infrastructure settings after 19.2 upgrades
What does this MR do and why?
Restores five AI infrastructure settings for Self-Managed instances affected by an upgrade to GitLab 19.2.
The regression was caused by the interaction of two migrations:
- !226045 (merged) adds
ai_settings.organization_idin a regular migration, then sets the existing singleton row to the default organization in a post-deploy migration. - !243574 (merged) copies the five settings from the default organization's
ai_settingsrow toapplication_settingsin a regular migration.
On Self-Managed upgrades, within milestone 19.2, all regular migrations run before post-deploy migrations. The copy therefore ran while the legacy row still had a NULL organization_id, found no default-organization row, and left the new columns at their defaults. This ordering applies whether or not SKIP_POST_DEPLOYMENT_MIGRATIONS is set; that variable only excludes post-deploy migrations from the current run. GitLab.com was unaffected because the migrations were deployed in separate deployment phases.
This post-deploy migration runs after the organization backfill and restores each setting from the preserved legacy row only when its destination still has the default introduced by the failed copy. This restores untouched or partially repaired installations without overwriting observable non-default administrator changes.
Caveat
If an administrator intentionally changed a setting to exactly its post-upgrade default, that state is indistinguishable from an unmigrated value using the preserved ai_settings and current application_settings rows. In that case, this migration restores the preserved legacy value. Non-default administrator changes are preserved.
Workaround
These workarounds are intended for affected GitLab 19.2 installations that cannot yet upgrade to a release containing this migration.
Diagnostic script and see values
Save the following as 606459-diagnose-duo-settings.rb on the GitLab instance. The script is read-only. It displays all five values from ai_settings and application_settings, then reports whether this migration issue is present and identifies values that were not migrated.
606459-diagnose-duo-settings.rb
# frozen_string_literal: true
SETTINGS = {
ai_gateway_url: [nil, ->(value) { value }],
ai_gateway_timeout_seconds: [60, ->(value) { value || 60 }],
enabled_instance_verbose_ai_logs: [false, ->(value) { !!value }],
duo_agent_platform_service_url: [nil, ->(value) { value }],
self_hosted_duo_agent_platform_service_secure: [true, ->(value) { value.nil? || value }]
}.freeze
legacy = Ai::Setting.find_by(organization_id: 1)
current = Gitlab::CurrentSettings.current_application_settings
abort 'Unable to diagnose: no default-organization ai_settings row was found.' unless legacy
abort 'Unable to diagnose: no application_settings row was found.' unless current
rows = SETTINGS.map do |attribute, (failed_copy_default, normalize)|
legacy_value = normalize.call(legacy.public_send(attribute))
current_value = current.public_send(attribute)
unmigrated = current_value == failed_copy_default && current_value != legacy_value
[attribute.to_s, legacy_value.inspect, current_value.inspect, unmigrated]
end
headers = ['Setting', 'ai_settings', 'application_settings']
widths = headers.each_index.map do |index|
([headers[index]] + rows.map { |row| row[index] }).map(&:length).max
end
format = widths.map { |width| "%-#{width}s" }.join(' | ')
puts format % headers
puts widths.map { |width| '-' * width }.join('-+-')
rows.each { |row| puts format % row.first(3) }
puts
unmigrated = rows.select(&:last)
if unmigrated.empty?
puts 'No evidence of unmigrated Duo settings was found.'
exit
end
puts 'This instance might have been affected by the migration issue.'
puts 'These settings still have the post-upgrade defaults and differ from their legacy values:'
puts unmigrated.map(&:first).join(', ')
puts 'Verify whether an administrator intentionally changed these settings after the upgrade.'Run the diagnostic script:
sudo gitlab-rails runner /path/to/606459-diagnose-duo-settings.rbThe script does not change any data. If it reports the issue, use the manual workaround below or upgrade to a release containing this migration.
Manual fix
In Admin area > GitLab Duo > Configuration, restore and save the affected service endpoint URLs. Also verify the AI Gateway timeout, verbose AI logging, and secure Duo Agent Platform connection settings against the values intended before the upgrade.
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps analyze changes for quality, performance, reliability, security, and maintainability.