Backend: Failed pipelines are not accessible when the logs are deleted on the object storage
Summary
One of our customers reported that they're receiving an Error 500 when accessing old failed pipelines. They don't experienced this with old pipelines that passed.
They are using Ceph for object storage with a retention policy of 10 days. I managed to reproduce the error when using Minio.
It seems that we are prefetching the job logs when accessing the pipeline? I'm still not sure why it doesn't happen on pipelines that passed, even if the job logs of that pipeline is also deleted.
Steps to reproduce
- Setup a GitLab instance and configure it to use an object storage.
- Run a pipeline that will fail.
- Delete the logs of the jobs related to the pipeline in the object storage.
- Access the page of the pipeline.
What is the current bug behavior?
GitLab shows an error 500.
What is the expected correct behavior?
GitLab should load the pipeline page
Relevant logs and/or screenshots
Logs when accessing a failed pipeline and the job logs are deleted:
"meta.project": "root/container-scanning",
"meta.root_namespace": "root",
"meta.caller_id": "Projects::PipelinesController#show",
...
"exception.class": "ActionView::Template::Error",
"exception.message": "Gitlab::HttpIO::FailedToGetChunkError",
"exception.backtrace": [
"lib/gitlab/http_io.rb:156:in `get_chunk'",
"lib/gitlab/http_io.rb:84:in `read'",
"lib/gitlab/ci/trace/stream.rb:143:in `read_backward'",
"lib/gitlab/ci/trace/stream.rb:127:in `reverse_line'",
"lib/gitlab/ci/trace/stream.rb:120:in `each'",
"lib/gitlab/ci/trace/stream.rb:120:in `first'",
"lib/gitlab/ci/trace/stream.rb:120:in `read_last_lines'",
"lib/gitlab/ci/trace/stream.rb:63:in `raw'",
"lib/gitlab/ci/trace/stream.rb:70:in `html'",
"lib/gitlab/ci/trace.rb:36:in `block in html'",
"lib/gitlab/ci/trace.rb:166:in `read_stream'",
"lib/gitlab/ci/trace.rb:89:in `read'",
"lib/gitlab/ci/trace.rb:35:in `html'",
"app/helpers/ci/builds_helper.rb:10:in `build_summary'",
"app/views/projects/pipelines/_with_tabs.html.haml:78",
"app/views/projects/pipelines/_with_tabs.html.haml:54:in `each_with_index'",
"app/views/projects/pipelines/_with_tabs.html.haml:54",
"app/views/projects/pipelines/show.html.haml:29",
"app/controllers/application_controller.rb:126:in `render'",
"app/controllers/projects/pipelines_controller.rb:223:in `block (2 levels) in render_show'",
"app/controllers/projects/pipelines_controller.rb:221:in `render_show'",
"app/controllers/projects/pipelines_controller.rb:101:in `block (2 levels) in show'",
"app/controllers/projects/pipelines_controller.rb:100:in `show'",
"app/controllers/application_controller.rb:536:in `block in allow_gitaly_ref_name_caching'",
"lib/gitlab/gitaly_client.rb:341:in `allow_ref_name_caching'",
"app/controllers/application_controller.rb:535:in `allow_gitaly_ref_name_caching'",
"ee/lib/gitlab/ip_address_state.rb:10:in `with'",
"ee/app/controllers/ee/application_controller.rb:44:in `set_current_ip_address'",
"app/controllers/application_controller.rb:485:in `set_current_admin'",
"lib/gitlab/session.rb:11:in `with_session'",
"app/controllers/application_controller.rb:476:in `set_session_storage'",
"lib/gitlab/i18n.rb:105:in `with_locale'",
"lib/gitlab/i18n.rb:111:in `with_user_locale'",
"app/controllers/application_controller.rb:470:in `set_locale'",
"app/controllers/application_controller.rb:464:in `set_current_context'",
"lib/gitlab/metrics/elasticsearch_rack_middleware.rb:16:in `call'",
"lib/gitlab/middleware/rails_queue_duration.rb:33:in `call'",
"lib/gitlab/metrics/rack_middleware.rb:16:in `block in call'",
"lib/gitlab/metrics/web_transaction.rb:21:in `run'",
"lib/gitlab/metrics/rack_middleware.rb:16:in `call'",
"lib/gitlab/middleware/speedscope.rb:13:in `call'",
"lib/gitlab/request_profiler/middleware.rb:17:in `call'",
"lib/gitlab/jira/middleware.rb:19:in `call'",
"lib/gitlab/middleware/go.rb:20:in `call'",
"lib/gitlab/etag_caching/middleware.rb:21:in `call'",
"lib/gitlab/middleware/multipart.rb:172:in `call'",
"lib/gitlab/middleware/read_only/controller.rb:50:in `call'",
"lib/gitlab/middleware/read_only.rb:18:in `call'",
"lib/gitlab/middleware/same_site_cookies.rb:27:in `call'",
"lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'",
"lib/gitlab/middleware/basic_health_check.rb:25:in `call'",
"lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'",
"lib/gitlab/middleware/request_context.rb:21:in `call'",
"config/initializers/fix_local_cache_middleware.rb:11:in `call'",
"lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call'",
"lib/gitlab/middleware/sidekiq_web_static.rb:20:in `call'",
"lib/gitlab/metrics/requests_rack_middleware.rb:74:in `call'",
"lib/gitlab/middleware/release_env.rb:12:in `call'"
],
Output of checks
Results of GitLab environment info
I managed to reproduce the problem on my test instance so I'm going to attach the information about my test instance instead:
Expand for output related to GitLab environment info
System information System: Proxy: no Current User: git Using RVM: no Ruby Version: 2.7.2p137 Gem Version: 3.2.19 Bundler Version:2.2.19 Rake Version: 13.0.3 Redis Version: unknown Git Version: unknown Sidekiq Version:5.2.9 Go Version: unknown GitLab information Version: 14.1.2-ee Revision: 0bf9b154ab4 Directory: /srv/gitlab DB Adapter: PostgreSQL DB Version: 12.7 URL: https://gitlab.gitlab-kubernetes.jpid.xyz HTTP Clone URL: https://gitlab.gitlab-kubernetes.jpid.xyz/some-group/some-project.git SSH Clone URL: git@gitlab.gitlab-kubernetes.jpid.xyz:some-group/some-project.git Elasticsearch: no Geo: no Using LDAP: no Using Omniauth: no GitLab Shell Version: 13.19.1 Repository storage paths: - default: /var/opt/gitlab/repo GitLab Shell path: /home/git/gitlab-shell Git: /usr/bin/git
Results of GitLab application Check
Expand for output related to the GitLab application check
Checking GitLab subtasks ...
Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 13.19.1 ? ... OK (13.19.1) Running /home/git/gitlab-shell/bin/check gitlab-shell self-check failed Try fixing it: Make sure GitLab is running; Check the gitlab-shell configuration file: sudo -u git -H editor /home/git/gitlab-shell/config.yml Please fix the error above and rerun the checks.
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... no Try fixing it: sudo -u git -H RAILS_ENV=production bin/background_jobs start For more information see: doc/install/installation.md in section "Install Init Script" see log/sidekiq.log for possible errors Please fix the error above and rerun the checks.
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Reply by email is disabled in config/gitlab.yml
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab App ...
Git configured correctly? ... no Trying to fix error automatically. ...Failed Try fixing it: sudo -u git -H "/usr/bin/git" config --global core.autocrlf "input" For more information see: doc/install/installation.md in section "GitLab" Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... skipped (no tmp uploads folder yet) Init script exists? ... no Try fixing it: Install the init script For more information see: doc/install/installation.md in section "Install Init Script" Please fix the error above and rerun the checks. Init script up-to-date? ... can't check because of previous errors Projects have namespace: ... 2/1 ... yes 1/2 ... yes 1/3 ... yes 6/4 ... yes 1/6 ... yes 1/7 ... yes 1/8 ... yes 1/9 ... yes 1/10 ... yes 16/11 ... yes Redis version >= 5.0.0? ... yes Ruby version >= 2.7.2 ? ... yes (2.7.2) Git version >= 2.31.0 ? ... no Your git bin path is "/usr/bin/git" Try fixing it: Update your git to a version >= 2.31.0 from Unknown Please fix the error above and rerun the checks. Git user has default SSH configuration? ... yes Active users: ... 5 Is authorized keys file accessible? ... skipped (authorized keys not enabled) GitLab configured to store new projects in hashed storage? ... yes All projects are in hashed storage? ... yes Elasticsearch version 7.x (6.4 - 6.x deprecated to be removed in 13.8)? ... skipped (elasticsearch is disabled)
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished
(we will only investigate if the tests are passing)