## Summary After upgrading a Self-Managed (Omnibus) GitLab instance to **19.2**, the GitLab Duo Self-Hosted service endpoint settings are cleared. Specifically, the following fields under **Admin area → GitLab Duo → Configuration → Service endpoints** become empty: - **Local AI Gateway URL** (used for GitLab Duo Self-Hosted requests) - **Local URL for the GitLab Duo Agent Platform service** (used for GitLab Duo Agent Platform requests) Because these URLs are cleared, GitLab Duo Self-Hosted features stop working entirely until the URLs are manually re-entered. ## Steps to reproduce 1. Have a Self-Managed (Omnibus) instance running **19.0** or **19.1** with GitLab Duo Self-Hosted configured, including: - **Local AI Gateway URL** - **Local URL for the GitLab Duo Agent Platform service** 2. Confirm GitLab Duo Self-Hosted features are working. 3. Upgrade the instance to **19.2**. 4. Navigate to **Admin area → GitLab Duo → Configuration → Service endpoints**. ## Current behavior - Both **Local AI Gateway URL** and **Local URL for the GitLab Duo Agent Platform service** fields are empty after the upgrade. - GitLab Duo Self-Hosted features do not work at all (nothing responds). ## Expected behavior - The previously configured **Local AI Gateway URL** and **Local URL for the GitLab Duo Agent Platform service** values should be preserved across the upgrade. - GitLab Duo Self-Hosted features should continue working after the upgrade without manual reconfiguration. ## Workaround These workarounds are intended for affected GitLab 19.2 installations that cannot yet upgrade to a release containing the fix. ### Diagnostic script and see values Save the following as `606459-diagnose-duo-settings.rb` on the GitLab instance. The script is read-only. It displays all five values from `ai_settings` and `application_settings`, then reports whether the instance might have been affected and identifies values that may not have been migrated. <details> <summary>606459-diagnose-duo-settings.rb</summary> ```ruby # frozen_string_literal: true SETTINGS = { ai_gateway_url: [nil, ->(value) { value }], ai_gateway_timeout_seconds: [60, ->(value) { value || 60 }], enabled_instance_verbose_ai_logs: [false, ->(value) { !!value }], duo_agent_platform_service_url: [nil, ->(value) { value }], self_hosted_duo_agent_platform_service_secure: [true, ->(value) { value.nil? || value }] }.freeze legacy = Ai::Setting.find_by(organization_id: 1) current = Gitlab::CurrentSettings.current_application_settings abort 'Unable to diagnose: no default-organization ai_settings row was found.' unless legacy abort 'Unable to diagnose: no application_settings row was found.' unless current rows = SETTINGS.map do |attribute, (failed_copy_default, normalize)| legacy_value = normalize.call(legacy.public_send(attribute)) current_value = current.public_send(attribute) unmigrated = current_value == failed_copy_default && current_value != legacy_value [attribute.to_s, legacy_value.inspect, current_value.inspect, unmigrated] end headers = ['Setting', 'ai_settings', 'application_settings'] widths = headers.each_index.map do |index| ([headers[index]] + rows.map { |row| row[index] }).map(&:length).max end format = widths.map { |width| "%-#{width}s" }.join(' | ') puts format % headers puts widths.map { |width| '-' * width }.join('-+-') rows.each { |row| puts format % row.first(3) } puts unmigrated = rows.select(&:last) if unmigrated.empty? puts 'No evidence of unmigrated Duo settings was found.' exit end puts 'This instance might have been affected by the migration issue.' puts 'These settings still have the post-upgrade defaults and differ from their legacy values:' puts unmigrated.map(&:first).join(', ') puts 'Verify whether an administrator intentionally changed these settings after the upgrade.' ``` </details> Run the diagnostic script: ```shell sudo gitlab-rails runner /path/to/606459-diagnose-duo-settings.rb ``` The script does not change any data. Because an administrator might have intentionally changed a setting to its default after the upgrade, its result indicates a possible match rather than proving that the migration issue occurred. ### Manual In **Admin area → GitLab Duo → Configuration**, restore and save the affected service endpoint URLs. Also verify the AI Gateway timeout, verbose AI logging, and secure Duo Agent Platform connection settings against the values intended before the upgrade. ## Environment - **GitLab version:** 19.2 (upgraded from 19.0 and 19.1 — reproduced from both source versions) - **Deployment type:** Self-Managed - **Installation method:** Omnibus - **Affected feature:** GitLab Duo Self-Hosted (Service endpoints configuration) ## Impact - Data loss of admin-configured settings during upgrade. - GitLab Duo Self-Hosted features are completely unavailable after upgrade until manual reconfiguration. ## Root cause The regression is caused by the interaction of two changes included in GitLab 19.2: - https://gitlab.com/gitlab-org/gitlab/-/merge_requests/226045 adds `ai_settings.organization_id` in a regular migration. Existing Self-Managed instances already have a singleton `ai_settings` row, so its new `organization_id` initially remains NULL. A later post-deploy migration assigns that row to the default organization (`organization_id = 1`). - https://gitlab.com/gitlab-org/gitlab/-/merge_requests/243574 moves five instance-level AI infrastructure settings from `ai_settings` to `application_settings`. Its regular data migration tries to copy values from the `ai_settings` row with `organization_id = 1`. On a Self-Managed upgrade, all regular migrations run before post-deploy migrations. The settings copy therefore runs before the existing `ai_settings` row is assigned to the default organization. It finds no row with `organization_id = 1`, returns without copying the data, and leaves these new `application_settings` columns at their defaults: - `ai_gateway_url`: NULL - `ai_gateway_timeout_seconds`: `60` - `enabled_instance_verbose_ai_logs`: `false` - `duo_agent_platform_service_url`: NULL - `self_hosted_duo_agent_platform_service_secure`: `true` The application reads these values from `application_settings` after the upgrade, while the original values remain preserved in `ai_settings`. This is why the endpoint URLs appear empty and other customized infrastructure settings may silently revert to their defaults. GitLab.com was not affected because its deployment process ran the relevant migration phases separately, allowing the organization backfill to complete before the settings copy ran. Self-Managed installations execute the regular and post-deploy migration batches during one upgrade, exposing the ordering dependency. The fix is tracked in https://gitlab.com/gitlab-org/gitlab/-/merge_requests/246616.