Job logs sometimes missing after run
Summary
After deploying the Gitlab on our Kubernetes cluster it works fine but some of the jobs do not show their logs anymore when their are finished. While it is still running the logs show up fine. When it is done sometimes the logs of the job cannot be shown anymore. When looked at it in the raw view it can be viewed. I have also looked into the storage bucket and the build logs are saved. It only does not show in the job screen. It loads for a long time and after a minute it gives an timeout. It also always happen on the same jobs, it is not completely random.
I have also added Gitlab to our Sentry server which gives the following error message:
Rack::Timeout::RequestTimeoutException: Request ran for longer than 60000ms
Steps to reproduce
Run a job, wait until it is done and look if the job log is there.
Configuration used
global:
edition: ce
hosts:
domain: domain
psql:
host: host
port: port
database: database
username: username
password:
secret: postgresql-password
key: postgresql-password
minio:
enabled: false
registry:
bucket: gitlab-registry-storage
appConfig:
lfs:
bucket: gitlab-lfs-storage
connection:
secret: objectstore
key: connection
artifacts:
bucket: gitlab-artifacts-storage
connection:
secret: objectstore
key: connection
uploads:
bucket: -gitlab-uploads-storage
connection:
secret: objectstore
key: connection
packages:
bucket: gitlab-packages-storage
connection:
secret: objectstore
key: connection
backups:
bucket: gitlab-backup-storage
tmpBucket: gitlab-tmp-storage
ldap:
preventSignin: true
servers:
main:
allow_username_or_email_login: true
label: 'LDAP'
host: host
port: 389
uid: 'uid'
bind_dn: binddn
base: base
group_base: group
password:
secret: ldap-password
key: password
omniauth:
enabled: true
allowSingleSignOn: ['saml']
ync_profile_from_provider: ['saml']
blockAutoCreatedUsers: false
autoLinkLdapUser: true
autoLinkSamlUser: false
providers:
- secret: saml
key: provider
sentry:
enabled: true
dsn: dns
clientside_dsn: dns
email:
display_name: Gitlab
from: email
smtp:
enabled: true
address: address
authentication: login
starttls_auto: true
port: 587
user_name: name
password:
key: smtp-password
secret: smtp-password
postgresql:
install: false
redis:
cluster:
enabled: true
gitlab:
task-runner:
backups:
cron:
enabled: true
schedule: 0 4 * * *
objectStorage:
config:
secret: objectstore-backups
key: config
registry:
storage:
secret: objectstore-registry
key: config
certmanager-issuer:
email: email
Current behavior
The job logs are sometimes not shown in the job window, it gives an error after waiting for a minute.
Expected behavior
The job logs to show.
Versions
- Chart: 4.1.1
- Platform:
- Cloud: Scaleway
- Kubernetes:
- Client: 1.18.2
- Server: 1.18.1
- Helm: (
helm version)- Server: 3.2.0
Relevant logs
The stacktrace as seen in Sentry:
Rack::Timeout::RequestTimeoutException: Request ran for longer than 60000ms
from net/protocol.rb:217:in `wait_readable'
from net/protocol.rb:217:in `rbuf_fill'
from net/protocol.rb:191:in `readuntil'
from net/protocol.rb:201:in `readline'
from net/http/response.rb:40:in `read_status_line'
from net/http/response.rb:29:in `read_new'
from seahorse/client/net_http/patches.rb:29:in `block in new_transport_request'
from seahorse/client/net_http/patches.rb:26:in `catch'
from seahorse/client/net_http/patches.rb:26:in `new_transport_request'
from net/http.rb:1479:in `request'
from lib/gitlab/http_io.rb:153:in `block in get_chunk'
from net/http.rb:920:in `start'
from net/http.rb:605:in `start'
from lib/gitlab/http_io.rb:152:in `get_chunk'
from lib/gitlab/http_io.rb:109:in `readline'
from lib/gitlab/http_io.rb:71:in `each_line'
from lib/gitlab/ci/ansi2json/converter.rb:28:in `convert'
from lib/gitlab/ci/ansi2json.rb:8:in `convert'
from app/models/ci/build_trace.rb:21:in `initialize'
from app/controllers/projects/jobs_controller.rb:62:in `new'
from app/controllers/projects/jobs_controller.rb:62:in `block (3 levels) in trace'
from action_controller/metal/mime_responds.rb:214:in `respond_to'
from app/controllers/projects/jobs_controller.rb:54:in `block in trace'
from lib/gitlab/ci/trace.rb:95:in `read'
from app/controllers/projects/jobs_controller.rb:53:in `trace'
from action_controller/metal/basic_implicit_render.rb:6:in `send_action'
from abstract_controller/base.rb:195:in `process_action'
from action_controller/metal/rendering.rb:30:in `process_action'
from abstract_controller/callbacks.rb:42:in `block in process_action'
from active_support/callbacks.rb:112:in `block in run_callbacks'
from app/controllers/application_controller.rb:497:in `set_current_admin'
from active_support/callbacks.rb:121:in `block in run_callbacks'
from lib/gitlab/session.rb:11:in `with_session'
from app/controllers/application_controller.rb:488:in `set_session_storage'
from active_support/callbacks.rb:121:in `block in run_callbacks'
from lib/gitlab/i18n.rb:55:in `with_locale'
from lib/gitlab/i18n.rb:61:in `with_user_locale'
from app/controllers/application_controller.rb:482:in `set_locale'
from active_support/callbacks.rb:121:in `block in run_callbacks'
from lib/gitlab/error_tracking.rb:51:in `with_context'
from app/controllers/application_controller.rb:547:in `sentry_context'
from active_support/callbacks.rb:121:in `block in run_callbacks'
from raven/integrations/rails/controller_transaction.rb:7:in `block in included'
from active_support/callbacks.rb:121:in `instance_exec'
from active_support/callbacks.rb:121:in `block in run_callbacks'
from marginalia.rb:111:in `record_query_comment'
from active_support/callbacks.rb:121:in `block in run_callbacks'
from app/controllers/application_controller.rb:475:in `block in set_current_context'
from lib/gitlab/application_context.rb:52:in `block in use'
from labkit/context.rb:32:in `with_context'
from lib/gitlab/application_context.rb:52:in `use'
from lib/gitlab/application_context.rb:20:in `with_context'
from app/controllers/application_controller.rb:468:in `set_current_context'
from active_support/callbacks.rb:121:in `block in run_callbacks'
from active_support/callbacks.rb:139:in `run_callbacks'
from abstract_controller/callbacks.rb:41:in `process_action'
from action_controller/metal/rescue.rb:22:in `process_action'
from action_controller/metal/instrumentation.rb:33:in `block in process_action'
from active_support/notifications.rb:180:in `block in instrument'
from active_support/notifications/instrumenter.rb:24:in `instrument'
from active_support/notifications.rb:180:in `instrument'
from action_controller/metal/instrumentation.rb:32:in `process_action'
from action_controller/metal/params_wrapper.rb:245:in `process_action'
from active_record/railties/controller_runtime.rb:27:in `process_action'
from abstract_controller/base.rb:136:in `process'
from action_view/rendering.rb:39:in `process'
from action_controller/metal.rb:190:in `dispatch'
from action_controller/metal.rb:254:in `dispatch'
from action_dispatch/routing/route_set.rb:50:in `dispatch'
from action_dispatch/routing/route_set.rb:33:in `serve'
from action_dispatch/routing/mapper.rb:18:in `block in <class:Constraints>'
from action_dispatch/routing/mapper.rb:48:in `serve'
from action_dispatch/journey/router.rb:49:in `block in serve'
from action_dispatch/journey/router.rb:32:in `each'
from action_dispatch/journey/router.rb:32:in `serve'
from action_dispatch/routing/route_set.rb:834:in `call'
from omniauth/strategy.rb:420:in `call_app!'
from omniauth/strategies/saml.rb:86:in `other_phase'
from omniauth/strategy.rb:190:in `call!'
from omniauth/strategy.rb:169:in `call'
from omniauth/strategy.rb:192:in `call!'
from omniauth/strategy.rb:169:in `call'
from lib/gitlab/metrics/elasticsearch_rack_middleware.rb:24:in `call'
from lib/gitlab/metrics/redis_rack_middleware.rb:22:in `call'
from lib/gitlab/middleware/rails_queue_duration.rb:29:in `call'
from lib/gitlab/metrics/rack_middleware.rb:17:in `block in call'
from lib/gitlab/metrics/transaction.rb:54:in `run'
from lib/gitlab/metrics/rack_middleware.rb:17:in `call'
from lib/gitlab/request_profiler/middleware.rb:17:in `call'
from labkit/middleware/rack.rb:19:in `block in call'
from labkit/context.rb:32:in `with_context'
from labkit/middleware/rack.rb:18:in `call'
from lib/gitlab/middleware/go.rb:20:in `call'
from lib/gitlab/etag_caching/middleware.rb:13:in `call'
from batch_loader/middleware.rb:11:in `call'
from rack/attack.rb:97:in `call'
from apollo_upload_server/middleware.rb:20:in `call'
from lib/gitlab/middleware/multipart.rb:125:in `call'
from rack/attack.rb:111:in `call'
from warden/manager.rb:36:in `block in call'
from warden/manager.rb:34:in `catch'
from warden/manager.rb:34:in `call'
from rack/cors.rb:98:in `call'
from rack/tempfile_reaper.rb:15:in `call'
from rack/etag.rb:25:in `call'
from rack/conditional_get.rb:25:in `call'
from rack/head.rb:12:in `call'
from action_dispatch/http/content_security_policy.rb:18:in `call'
from lib/gitlab/middleware/read_only/controller.rb:51:in `call'
from lib/gitlab/middleware/read_only.rb:18:in `call'
from rack/session/abstract/id.rb:259:in `context'
from rack/session/abstract/id.rb:253:in `call'
from action_dispatch/middleware/cookies.rb:648:in `call'
from lib/gitlab/middleware/same_site_cookies.rb:27:in `call'
from action_dispatch/middleware/callbacks.rb:27:in `block in call'
from active_support/callbacks.rb:101:in `run_callbacks'
from action_dispatch/middleware/callbacks.rb:26:in `call'
from action_dispatch/middleware/actionable_exceptions.rb:17:in `call'
from action_dispatch/middleware/debug_exceptions.rb:32:in `call'
from action_dispatch/middleware/show_exceptions.rb:33:in `call'
from lib/gitlab/middleware/basic_health_check.rb:25:in `call'
from rails/rack/logger.rb:37:in `call_app'
from rails/rack/logger.rb:26:in `block in call'
from active_support/tagged_logging.rb:80:in `block in tagged'
from active_support/tagged_logging.rb:28:in `tagged'
from active_support/tagged_logging.rb:80:in `tagged'
from rails/rack/logger.rb:26:in `call'
from action_dispatch/middleware/remote_ip.rb:81:in `call'
from lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'
from lib/gitlab/middleware/request_context.rb:23:in `call'
from request_store/middleware.rb:19:in `call'
from action_dispatch/middleware/request_id.rb:27:in `call'
from rack/method_override.rb:22:in `call'
from rack/runtime.rb:22:in `call'
from rack/timeout/core.rb:123:in `block in call'
from rack/timeout/support/timeout.rb:19:in `timeout'
from rack/timeout/core.rb:122:in `call'
from config/initializers/fix_local_cache_middleware.rb:9:in `call'
from action_dispatch/middleware/executor.rb:14:in `call'
from rack/sendfile.rb:111:in `call'
from action_dispatch/middleware/host_authorization.rb:76:in `call'
from lib/gitlab/metrics/requests_rack_middleware.rb:60:in `call'
from raven/integrations/rack.rb:51:in `call'
from rails/engine.rb:527:in `call'
from rails/railtie.rb:190:in `public_send'
from rails/railtie.rb:190:in `method_missing'
from lib/gitlab/middleware/release_env.rb:12:in `call'
from rack/urlmap.rb:68:in `block in call'
from rack/urlmap.rb:53:in `each'
from rack/urlmap.rb:53:in `call'
from puma/configuration.rb:228:in `call'
from puma/server.rb:691:in `handle_request'
from puma/server.rb:481:in `process_client'
from puma/server.rb:335:in `block in run'
from puma/thread_pool.rb:138:in `block in spawn_thread'