GitLab Duo Self-hosted features not working after upgrading to v18.4

Summary

Following the upgrade to GitLab v18.4, Customers lost complete access to Duo Self-Hosted model configuration & features due to GraphQL errors

Steps to reproduce

• Upgrade to GitLab 18.4.0
• Navigate to Admin → GitLab Duo → Configure Duo Self-Hosted → AI-native features which displays An error occurred while loading the AI feature settings. Please try again.
• Model configuration fails with ERROR: The self-hosted model server returned code 403: Forbidden
• All Duo self-hosted features are not functional

Example Project

What is the current bug behavior?

Failure to load model configuration interface with GraphQL 500 errors.

What is the expected correct behavior?

Model configuration UI loads successfully and features should work as expected

Relevant logs and/or screenshots

• GraphQL Error: getAiFeatureSettings query returns 500 FrozenError when accessing AI-native features page
• Stack Trace: ee/app/models/ai/feature_setting.rb:79 attempting to modify frozen hash
• VS Code Extension Logs: Direct connection failures with 503 "Token creation failed" errors to /api/v4/code_suggestions/direct_access
• Network Telemetry: Confirms VS Code falling back from direct to indirect connections successfully

"correlation_id":"01K5G0PNYDNDF99TG3NXYJPYNZ",
"meta.caller_id":"GraphqlController#execute",
"meta.feature_category":"self-hosted_models",
"meta.organization_id":1,
"meta.remote_ip":"xxxxx",
"meta.user":"root",
"meta.user_id":1,
"meta.client_id":"user/1",
"exception.class":"NoMethodError",
"exception.message":"undefined method `payload' for nil:NilClass",
"exception.backtrace":[
   "ee/app/graphql/resolvers/ai/feature_settings/feature_settings_resolver.rb:43:in `gitlab_model_definitions'",
   "ee/app/graphql/resolvers/ai/feature_settings/feature_settings_resolver.rb:24:in `resolve'",
   "graphql (2.5.11) lib/graphql/schema/resolver.rb:120:in `public_send'",
   "graphql (2.5.11) lib/graphql/schema/resolver.rb:120:in `call_resolve'",
   "graphql (2.5.11) lib/graphql/schema/resolver.rb:105:in `block (3 levels) in resolve_with_support'",
   "graphql (2.5.11) lib/graphql/schema.rb:1647:in `after_lazy'",
   "graphql (2.5.11) lib/graphql/query.rb:30:in `after_lazy'",
   "graphql (2.5.11) lib/graphql/schema/resolver.rb:93:in `block (2 levels) in resolve_with_support'",
   "graphql (2.5.11) lib/graphql/schema.rb:1647:in `after_lazy'",
   "graphql (2.5.11) lib/graphql/query.rb:30:in `after_lazy'",
   "graphql (2.5.11) lib/graphql/schema/resolver.rb:84:in `block in resolve_with_support'",
   "graphql (2.5.11) lib/graphql/schema.rb:1647:in `after_lazy'",
   "graphql (2.5.11) lib/graphql/query.rb:30:in `after_lazy'",
   "graphql (2.5.11) lib/graphql/schema/resolver.rb:72:in `resolve_with_support'",
   "graphql (2.5.11) lib/graphql/schema/field.rb:760:in `public_send'",
   "graphql (2.5.11) lib/graphql/schema/field.rb:760:in `block (2 levels) in resolve'",
   "graphql (2.5.11) lib/graphql/schema/field.rb:903:in `block in with_extensions'",
   "graphql (2.5.11) lib/graphql/schema/field.rb:939:in `block (2 levels) in run_extensions_before_resolve'",
   "graphql (2.5.11) lib/graphql/schema/field.rb:939:in `block (2 levels) in run_extensions_before_resolve'",
   "graphql (2.5.11) lib/graphql/schema/field.rb:939:in `block (2 levels) in run_extensions_before_resolve'",
   "graphql (2.5.11) lib/graphql/schema/field.rb:939:in `block (2 levels) in run_extensions_before_resolve'",
   "graphql (2.5.11) lib/graphql/schema/field.rb:942:in `run_extensions_before_resolve'",
   "graphql (2.5.11) lib/graphql/schema/field.rb:939:in `block in run_extensions_before_resolve'",
   "graphql (2.5.11) lib/graphql/schema/field_extension.rb:134:in `resolve'",
   "graphql (2.5.11) lib/graphql/schema/field.rb:926:in `run_extensions_before_resolve'",
   "graphql (2.5.11) lib/graphql/schema/field.rb:939:in `block in run_extensions_before_resolve'",
   "lib/gitlab/graphql/present/field_extension.rb:18:in `resolve'",
   "graphql (2.5.11) lib/graphql/schema/field.rb:926:in `run_extensions_before_resolve'",
   "graphql (2.5.11) lib/graphql/schema/field.rb:939:in `block in run_extensions_before_resolve'",
   "graphql (2.5.11) lib/graphql/schema/field/connection_extension.rb:21:in `resolve'",
   "graphql (2.5.11) lib/graphql/schema/field.rb:926:in `run_extensions_before_resolve'",
   "graphql (2.5.11) lib/graphql/schema/field.rb:939:in `block in run_extensions_before_resolve'",
   "graphql (2.5.11) lib/graphql/schema/field_extension.rb:134:in `resolve'",
   "graphql (2.5.11) lib/graphql/schema/field.rb:926:in `run_extensions_before_resolve'",
   "graphql (2.5.11) lib/graphql/schema/field.rb:898:in `with_extensions'",
   "graphql (2.5.11) lib/graphql/schema/field.rb:729:in `block in resolve'",
   "graphql (2.5.11) lib/graphql/schema.rb:1647:in `after_lazy'",
   "graphql (2.5.11) lib/graphql/query.rb:30:in `after_lazy'",
   "graphql (2.5.11) lib/graphql/schema/field.rb:727:in `resolve'",
   "graphql (2.5.11) lib/graphql/execution/interpreter/runtime.rb:462:in `block (2 levels) in evaluate_selection_with_resolved_keyword_args'",
   "graphql (2.5.11) lib/graphql/tracing/trace.rb:90:in `execute_field'",
   "\"batch-loader (2.0.5) lib/batch_loader/graphql.rb"

Output of checks

Results of GitLab environment info

Expand for output related to GitLab environment info

(For installations with omnibus-gitlab package run and paste the output of:
`sudo gitlab-rake gitlab:env:info`)

(For installations from source run and paste the output of:
`sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production`)

Results of GitLab application Check

Expand for output related to the GitLab application check

(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:check SANITIZE=true)
(For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true)
(we will only investigate if the tests are passing)

Workaround

Self-managed & Duo self-hosted customers who do not currently require the configuration of GitLab-provided models may revert to the previous system behavior and resolve this error by disabling the associated feature flag as mentioned here

Feature.disable(:instance_level_model_selection)

Possible fixes

Patch release information for backports

If the bug fix needs to be backported in a patch release to a version under the maintenance policy, please follow the steps on the patch release runbook for GitLab engineers.

Refer to the internal "Release Information" dashboard for information about the next patch release, including the targeted versions, expected release date, and current status.

High-severity bug remediation

To remediate high-severity issues requiring an internal release for single-tenant SaaS instances, refer to the internal release process for engineers.