Backend: An error occurred while fetching the job and viewing Job Log JSON, navigating to Settings -> Integrations produces 500 error after upgrade to v14.9.1
Summary
After upgrade to v14.9.1, I'm unable to open Settings -> Integrations on any project. It always produces a 500 error. I'm unable to retrieve .json logs for jobs without the environment keyword.
Steps to reproduce
Open Settings -> Integrations
Example Project
What is the current bug behavior?
Attempting to view an impacted job (only some jobs are impacted, no pattern has emerged yet, this happens with jobs that do not have the environment
keyword):
- "An error occurred while fetching the job."
Get 500 error:
- when browsing to Settings > Integrations
- when attempting to view info about an impacted job in JSON form (e.g.
/root/mr-pipeline/-/jobs/386.json
) for jobs (including those without the environment keyword).
What is the expected correct behavior?
- The job log opens without the "An error occurred while fetching the job." error.
- The corresponding JSON for that job is available without the
500
error.
Able to open Settings -> Integrations.
The same behaviour and error (similar stack trace) happens opening a Job trace when Environment name is set. If I disable environment field in gitlab-ci during a deploy job, then I can get job trace. An error occurred while fetching the job.
Relevant logs and/or screenshots
*** /var/log/gitlab/production.log ***
Started GET "XXXXXXXX/-/jobs/1171071.json" for XXXXXX at 2022-03-30 18:50:39 +0000
Processing by Projects::JobsController#show as JSON
Parameters: {"namespace_id"=>"XXXXX", "project_id"=>"XXXXXX", "id"=>"1171071"}
Completed 500 Internal Server Error in 496ms (ActiveRecord: 160.5ms | Elasticsearch: 0.0ms | Allocations: 63420)
NotImplementedError (NotImplementedError):
app/models/integration.rb:212:in `to_param'
app/models/integration.rb:424:in `to_param'
app/models/project.rb:2862:in `block in find_integration'
app/models/project.rb:2862:in `find_integration'
app/models/project.rb:1474:in `find_or_initialize_integration'
lib/gitlab/prometheus/adapter.rb:29:in `service_prometheus_adapter'
lib/gitlab/prometheus/adapter.rb:14:in `prometheus_adapter'
app/models/environment.rb:340:in `prometheus_adapter'
app/models/environment.rb:314:in `has_metrics?'
app/serializers/environment_entity.rb:30:in `block in <class:EnvironmentEntity>'
app/serializers/base_serializer.rb:16:in `represent'
app/controllers/projects/jobs_controller.rb:50:in `block (2 levels) in show'
app/controllers/projects/jobs_controller.rb:43:in `show'
ee/lib/gitlab/ip_address_state.rb:10:in `with'
ee/app/controllers/ee/application_controller.rb:45:in `set_current_ip_address'
app/controllers/application_controller.rb:499:in `set_current_admin'
lib/gitlab/session.rb:11:in `with_session'
app/controllers/application_controller.rb:490:in `set_session_storage'
lib/gitlab/i18n.rb:105:in `with_locale'
lib/gitlab/i18n.rb:111:in `with_user_locale'
app/controllers/application_controller.rb:484:in `set_locale'
app/controllers/application_controller.rb:478:in `set_current_context'
lib/gitlab/metrics/elasticsearch_rack_middleware.rb:16:in `call'
lib/gitlab/middleware/rails_queue_duration.rb:33:in `call'
lib/gitlab/middleware/memory_report.rb:13:in `call'
lib/gitlab/middleware/speedscope.rb:13:in `call'
lib/gitlab/request_profiler/middleware.rb:17:in `call'
lib/gitlab/database/load_balancing/rack_middleware.rb:23:in `call'
lib/gitlab/metrics/rack_middleware.rb:16:in `block in call'
lib/gitlab/metrics/web_transaction.rb:46:in `run'
lib/gitlab/metrics/rack_middleware.rb:16:in `call'
lib/gitlab/jira/middleware.rb:19:in `call'
lib/gitlab/middleware/go.rb:20:in `call'
lib/gitlab/etag_caching/middleware.rb:21:in `call'
lib/gitlab/middleware/query_analyzer.rb:11:in `block in call'
lib/gitlab/database/query_analyzer.rb:46:in `within'
lib/gitlab/middleware/query_analyzer.rb:11:in `call'
lib/gitlab/middleware/multipart.rb:173:in `call'
lib/gitlab/middleware/read_only/controller.rb:50:in `call'
lib/gitlab/middleware/read_only.rb:18:in `call'
lib/gitlab/middleware/same_site_cookies.rb:27:in `call'
lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'
lib/gitlab/middleware/basic_health_check.rb:25:in `call'
lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'
lib/gitlab/middleware/request_context.rb:21:in `call'
lib/gitlab/middleware/webhook_recursion_detection.rb:15:in `call'
config/initializers/fix_local_cache_middleware.rb:11:in `call'
lib/gitlab/middleware/compressed_json.rb:26:in `call'
lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call'
lib/gitlab/middleware/sidekiq_web_static.rb:20:in `call'
lib/gitlab/metrics/requests_rack_middleware.rb:77:in `call'
lib/gitlab/middleware/release_env.rb:13:in `call'
Output of checks
Results of GitLab environment info
Expand for output related to GitLab environment info
(For installations with omnibus-gitlab package run and paste the output of: `sudo gitlab-rake gitlab:env:info`) (For installations from source run and paste the output of: `sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production`)
Results of GitLab application Check
Expand for output related to the GitLab application check
(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:check SANITIZE=true
)(For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true
)(we will only investigate if the tests are passing)
Possible fixes
Running the migration again fixes the issue: #356895 (comment 908647381)
Workarounds
Viewing Job Logs
For a job that shows "An error occurred while fetching the job" when viewing it and has the 500
when attempting to view the the job's JSON data, the raw job log can be viewed.
-
500
: https://gitlab.example.com/group/project/-/jobs/1337.json -
200
: https://gitlab.example.com/group/project/-/jobs/1337/raw
Removing the environment
keyword from that job also seems to resolve this problem for future job logs.