Kubernetes agent remote development "/prerequisites" unhandled NoMethodError

MR - Handle agent config not found gracefully (!196209 - closed)

MR - Handle agent config not found gracefully (!196223 - merged)

Summary

When enabling remote_development module for a Kubernetes agent, the resulting request to GET /api/:version/internal/kubernetes/modules/remote_development/prerequisites fails with 500 status code.

When I check GitLab.com logs for Rails, we see this started occurring around 23rd June 2025 ~ 03:00 UTC.

In the agent logs:

ime":"2025-06-27T04:45:21.819692444Z","level":"INFO","msg":"Starting","mod_name":"remote_development","agent_key":"agentk:2140838"}
{"time":"2025-06-27T04:45:24.369318259Z","level":"ERROR","msg":"Error polling GitLab for prerequisites","mod_name":"remote_development","agent_key":"agentk:2140838","error":"unexpected status code: 500"}
{"time":"2025-06-27T04:46:02.351431725Z","level":"ERROR","msg":"Error polling GitLab for prerequisites","mod_name":"remote_development","agent_key":"agentk:2140838","error":"unexpected status code: 500"}
{"time":"2025-06-27T04:47:48.578768494Z","level":"ERROR","msg":"Error polling GitLab for prerequisites","mod_name":"remote_development","agent_key":"agentk:2140838","error":"unexpected status code: 500"}

In GitLab Rails log:

        "class": "NoMethodError",
        "message": "undefined method `shared_namespace' for nil:NilClass"

Steps to reproduce

Create, install, register, and configure a Kubernetes agent
Enable remote_development module in the agent config:
```
remote_development:
 enabled: true
```
Check agent pod logs

Example Project

What is the current bug behavior?

API call from agent results in unhandled 500 error

What is the expected correct behavior?

API call from agent doesn't result in an unhandled error

Relevant logs and/or screenshots

https://log.gprd.gitlab.net/app/r/s/ZBfkw

Rails index pattern with search terms:

json.path:"/prerequisites" AND json.status:500

Backtrace:

      "exception": {
        "backtrace": [
          "ee/lib/remote_development/agent_prerequisites_operations/response_builder.rb:15:in `build'",
          "lib/gitlab/fp/result.rb:155:in `call'",
          "lib/gitlab/fp/result.rb:155:in `map'",
          "ee/lib/remote_development/agent_prerequisites_operations/main.rb:17:in `main'",
          "ee/app/services/remote_development/common_service.rb:34:in `call'",
          "ee/app/services/remote_development/common_service.rb:34:in `execute'",
          "ee/lib/ee/api/internal/kubernetes.rb:90:in `block (5 levels) in <module:Kubernetes>'",
          "lib/gitlab/middleware/ip_address.rb:11:in `call'",
          "lib/api/api_guard.rb:266:in `call'",
          "config/initializers/action_dispatch_journey_router.rb:52:in `block in find_routes'",
          "config/initializers/action_dispatch_journey_router.rb:25:in `map!'",
          "config/initializers/action_dispatch_journey_router.rb:25:in `find_routes'",
          "ee/lib/omni_auth/strategies/group_saml.rb:41:in `other_phase'",
          "lib/gitlab/metrics/elasticsearch_rack_middleware.rb:16:in `call'",
          "lib/gitlab/middleware/sidekiq_shard_awareness_validation.rb:20:in `block in call'",
          "lib/gitlab/sidekiq_sharding/validator.rb:42:in `enabled'",
          "lib/gitlab/middleware/sidekiq_shard_awareness_validation.rb:20:in `call'",
          "lib/gitlab/middleware/memory_report.rb:13:in `call'",
          "lib/gitlab/middleware/speedscope.rb:13:in `call'",
          "lib/gitlab/database/load_balancing/rack_middleware.rb:23:in `call'",
          "lib/gitlab/middleware/rails_queue_duration.rb:33:in `call'",
          "lib/gitlab/etag_caching/middleware.rb:21:in `call'",
          "lib/gitlab/metrics/rack_middleware.rb:16:in `block in call'",
          "lib/gitlab/metrics/web_transaction.rb:46:in `run'",
          "lib/gitlab/metrics/rack_middleware.rb:16:in `call'",
          "lib/gitlab/middleware/go.rb:21:in `call'",
          "lib/gitlab/middleware/query_analyzer.rb:11:in `block in call'",
          "lib/gitlab/database/query_analyzer.rb:83:in `within'",
          "lib/gitlab/middleware/query_analyzer.rb:11:in `call'",
          "lib/ci/job_token/middleware.rb:11:in `call'",
          "lib/gitlab/middleware/multipart.rb:173:in `call'",
          "lib/gitlab/middleware/read_only/controller.rb:50:in `call'",
          "lib/gitlab/middleware/read_only.rb:18:in `call'",
          "lib/gitlab/middleware/unauthenticated_session_expiry.rb:18:in `call'",
          "lib/gitlab/middleware/secure_headers.rb:11:in `call'",
          "lib/gitlab/middleware/same_site_cookies.rb:27:in `call'",
          "lib/gitlab/middleware/path_traversal_check.rb:40:in `call'",
          "lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'",
          "lib/gitlab/middleware/basic_health_check.rb:25:in `call'",
          "lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'",
          "lib/gitlab/middleware/request_context.rb:15:in `call'",
          "lib/gitlab/middleware/webhook_recursion_detection.rb:15:in `call'",
          "config/initializers/fix_local_cache_middleware.rb:11:in `call'",
          "lib/gitlab/middleware/compressed_json.rb:44:in `call'",
          "lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call'",
          "lib/gitlab/metrics/requests_rack_middleware.rb:83:in `call'",
          "lib/gitlab/middleware/sidekiq_web_static.rb:20:in `call'",
          "lib/gitlab/middleware/release_env.rb:12:in `call'"
        ],

Output of checks

Results of GitLab environment info

Expand for output related to GitLab environment info


(For installations with omnibus-gitlab package run and paste the output of:
`sudo gitlab-rake gitlab:env:info`)

(For installations from source run and paste the output of:
`sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production`)

Results of GitLab application Check

Expand for output related to the GitLab application check

(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:check SANITIZE=true)
(For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true)
(we will only investigate if the tests are passing)

Possible fixes

Patch release information for backports

If the bug fix needs to be backported in a patch release to a version under the maintenance policy, please follow the steps on the patch release runbook for GitLab engineers.

Refer to the internal "Release Information" dashboard for information about the next patch release, including the targeted versions, expected release date, and current status.

High-severity bug remediation

To remediate high-severity issues requiring an internal release for single-tenant SaaS instances, refer to the internal release process for engineers.

Edited Jul 01, 2025 by zli