Accessing the repository home results in a 500 error
I have already made an issue on this topic in the gitlab repo, but I think it is a gitaly problem, so I create this issue referring to the one in the gitlab repo.
Summary
I updated my gitlab to version 14.5, now I can't access the individual repositories. The gitlab starts, but as soon as I want to view a repository I get an internal server error. The Gitlab is running on a Rasperry Pi 4.
Logs
==> /var/log/gitlab/gitaly/current <==
{"correlation_id":"01FN8MX7S2W8HNZ40KPRZPADTR","grpc.code":"OK","grpc.meta.auth_version":"v2","grpc.meta.client_name":"gitlab-web","grpc.meta.deadline_type":"regular","grpc.meta.method_type":"unary","grpc.method":"RepositoryExists","grpc.request.deadline":"2021-11-24T10:37:37.006","grpc.request.fullMethod":"/gitaly.RepositoryService/RepositoryExists","grpc.request.glProjectPath":"deliverosaurus/server/menu-service","grpc.request.glRepository":"project-20","grpc.request.payload_bytes":140,"grpc.request.repoPath":"@hashed/f5/ca/f5ca38f748a1d6eaf726b8a42fb575c3c71f1864a8143301782de13da2d9202b.git","grpc.request.repoStorage":"HDD","grpc.response.payload_bytes":2,"grpc.service":"gitaly.RepositoryService","grpc.start_time":"2021-11-24T10:37:27.187","grpc.time_ms":17.905,"level":"info","msg":"finished unary call with code OK","peer.address":"@","pid":7014,"remote_ip":"84.147.40.175","span.kind":"server","system":"grpc","time":"2021-11-24T09:37:27.205Z","username":"morzan1001"}
{"error":"unaligned 64-bit atomic operation","level":"error","method":"/gitaly.CommitService/FindCommit","msg":"grpc panic","time":"2021-11-24T09:37:27.235Z"}
{"correlation_id":"01FN8MX7S2W8HNZ40KPRZPADTR","error":"rpc error: code = Internal desc = panic: unaligned 64-bit atomic operation","grpc.code":"Internal","grpc.meta.auth_version":"v2","grpc.meta.client_name":"gitlab-web","grpc.meta.deadline_type":"regular","grpc.meta.method_type":"unary","grpc.method":"FindCommit","grpc.request.deadline":"2021-11-24T10:37:57.009","grpc.request.fullMethod":"/gitaly.CommitService/FindCommit","grpc.request.glProjectPath":"deliverosaurus/server/menu-service","grpc.request.glRepository":"project-20","grpc.request.payload_bytes":146,"grpc.request.repoPath":"@hashed/f5/ca/f5ca38f748a1d6eaf726b8a42fb575c3c71f1864a8143301782de13da2d9202b.git","grpc.request.repoStorage":"HDD","grpc.response.payload_bytes":0,"grpc.service":"gitaly.CommitService","grpc.start_time":"2021-11-24T10:37:27.209","grpc.time_ms":25.6,"level":"error","msg":"finished unary call with code Internal","peer.address":"@","pid":7014,"remote_ip":"84.147.40.175","span.kind":"server","system":"grpc","time":"2021-11-24T09:37:27.235Z","username":"morzan1001"}
panic: unaligned 64-bit atomic operation
goroutine 169 [running]:
runtime/internal/atomic.panicUnaligned()
/usr/local/go/src/runtime/internal/atomic/unaligned.go:8 +0x24
runtime/internal/atomic.Load64(0x4868c14, 0x0, 0x0)
/usr/local/go/src/runtime/internal/atomic/asm_arm.s:263 +0x14
gitlab.com/gitlab-org/gitaly/v14/internal/git/catfile.(*requestQueue).isDirty(0x4868c08, 0x0)
/var/cache/omnibus/src/gitaly/internal/git/catfile/request_queue.go:60 +0x78
gitlab.com/gitlab-org/gitaly/v14/internal/git/catfile.(*objectReader).isDirty(0x4868c00, 0x0)
/var/cache/omnibus/src/gitaly/internal/git/catfile/object_reader.go:108 +0x24
gitlab.com/gitlab-org/gitaly/v14/internal/git/catfile.(*ProcessCache).returnWhenDone(0x23bfb60, 0x709c140, 0x23bfb70, 0x4ecf080, 0x24, 0x4b7c4a8, 0x3, 0x7050300, 0x52, 0x0, ...)
/var/cache/omnibus/src/gitaly/internal/git/catfile/cache.go:311 +0x78
created by gitlab.com/gitlab-org/gitaly/v14/internal/git/catfile.(*ProcessCache).getOrCreateProcess
/var/cache/omnibus/src/gitaly/internal/git/catfile/cache.go:277 +0x35c
{"gitaly":7014,"level":"warning","msg":"forwarding signal","pid":7008,"signal":17,"time":"2021-11-24T09:37:27.260Z","wrapper":7008}
{"error":"os: process already finished","gitaly":7014,"level":"error","msg":"can't forward the signal","pid":7008,"signal":17,"time":"2021-11-24T09:37:27.260Z","wrapper":7008}
A bit further down:
==> /var/log/gitlab/gitlab-rails/production.log <==
Rendered layout layouts/project.html.haml (Duration: 275.1ms | Allocations: 24743)
Completed 500 Internal Server Error in 439ms (ActiveRecord: 48.0ms | Elasticsearch: 0.0ms | Allocations: 41042)
==> /var/log/gitlab/gitlab-rails/production_json.log <==
{"method":"GET","path":"/morzan1001/Abschlussarbeit","format":"html","controller":"ProjectsController","action":"show","status":500,"time":"2021-11-24T09:37:30.824Z","params":[{"key":"namespace_id","value":"morzan1001"},{"key":"id","value":"Abschlussarbeit"}],"correlation_id":"01FN8MXB28D46YKS2CFFADFDVA","meta.user":"morzan1001","meta.project":"morzan1001/Abschlussarbeit","meta.root_namespace":"morzan1001","meta.caller_id":"ProjectsController#show","meta.remote_ip":"84.147.40.175","meta.feature_category":"projects","meta.client_id":"user/1","remote_ip":"84.147.40.175","user_id":1,"username":"morzan1001","ua":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:94.0) Gecko/20100101 Firefox/94.0","queue_duration_s":0.124608,"request_urgency":"default","target_duration_s":1,"gitaly_calls":1,"gitaly_duration_s":0.005798,"redis_calls":26,"redis_duration_s":0.01744,"redis_read_bytes":3707,"redis_write_bytes":2822,"redis_cache_calls":23,"redis_cache_duration_s":0.012682,"redis_cache_read_bytes":3526,"redis_cache_write_bytes":1322,"redis_shared_state_calls":3,"redis_shared_state_duration_s":0.004758,"redis_shared_state_read_bytes":181,"redis_shared_state_write_bytes":1500,"db_count":35,"db_write_count":0,"db_cached_count":7,"db_replica_count":0,"db_replica_cached_count":0,"db_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_count":35,"db_primary_cached_count":7,"db_primary_wal_count":0,"db_primary_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.085,"cpu_s":0.463385,"mem_objects":55765,"mem_bytes":3615318,"mem_mallocs":15212,"mem_total_bytes":4953678,"pid":1260,"exception.class":"ActionView::Template::Error","exception.message":"14:failed to connect to all addresses.","exception.backtrace":["lib/gitlab/git/wraps_gitaly_errors.rb:15:in `rescue in wrapped_gitaly_errors'","lib/gitlab/git/wraps_gitaly_errors.rb:6:in `wrapped_gitaly_errors'","lib/gitlab/git/repository.rb:148:in `branch_count'","app/models/repository.rb:569:in `branch_count'","lib/gitlab/repository_cache_adapter.rb:21:in `block (2 levels) in cache_method'","lib/gitlab/repository_cache.rb:25:in `fetch'","lib/gitlab/repository_cache_adapter.rb:163:in `block in cache_method_output'","lib/gitlab/utils/strong_memoize.rb:30:in `strong_memoize'","lib/gitlab/repository_cache_adapter.rb:203:in `block in memoize_method_output'","lib/gitlab/repository_cache_adapter.rb:212:in `no_repository_fallback'","lib/gitlab/repository_cache_adapter.rb:202:in `memoize_method_output'","lib/gitlab/repository_cache_adapter.rb:162:in `cache_method_output'","lib/gitlab/repository_cache_adapter.rb:20:in `block in cache_method'","app/presenters/project_presenter.rb:229:in `branches_anchor_data'","app/presenters/project_presenter.rb:30:in `statistics_anchors'","app/views/projects/_home_panel.html.haml:51","app/views/projects/_home_panel.html.haml:46","app/views/projects/show.html.haml:14","app/controllers/application_controller.rb:136:in `render'","app/controllers/application_controller.rb:555:in `block in allow_gitaly_ref_name_caching'","lib/gitlab/gitaly_client.rb:322:in `allow_ref_name_caching'","app/controllers/application_controller.rb:554:in `allow_gitaly_ref_name_caching'","app/controllers/application_controller.rb:504:in `set_current_admin'","lib/gitlab/session.rb:11:in `with_session'","app/controllers/application_controller.rb:495:in `set_session_storage'","lib/gitlab/i18n.rb:105:in `with_locale'","lib/gitlab/i18n.rb:111:in `with_user_locale'","app/controllers/application_controller.rb:489:in `set_locale'","app/controllers/application_controller.rb:483:in `set_current_context'","lib/gitlab/metrics/elasticsearch_rack_middleware.rb:16:in `call'","lib/gitlab/middleware/rails_queue_duration.rb:33:in `call'","lib/gitlab/middleware/speedscope.rb:13:in `call'","lib/gitlab/request_profiler/middleware.rb:17:in `call'","lib/gitlab/database/load_balancing/rack_middleware.rb:23:in `call'","lib/gitlab/metrics/rack_middleware.rb:16:in `block in call'","lib/gitlab/metrics/web_transaction.rb:46:in `run'","lib/gitlab/metrics/rack_middleware.rb:16:in `call'","lib/gitlab/jira/middleware.rb:19:in `call'","lib/gitlab/middleware/go.rb:20:in `call'","lib/gitlab/etag_caching/middleware.rb:21:in `call'","lib/gitlab/middleware/multipart.rb:173:in `call'","lib/gitlab/middleware/read_only/controller.rb:50:in `call'","lib/gitlab/middleware/read_only.rb:18:in `call'","lib/gitlab/middleware/same_site_cookies.rb:27:in `call'","lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'","lib/gitlab/middleware/basic_health_check.rb:25:in `call'","lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'","lib/gitlab/middleware/request_context.rb:21:in `call'","config/initializers/fix_local_cache_middleware.rb:11:in `call'","lib/gitlab/middleware/compressed_json.rb:26:in `call'","lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call'","lib/gitlab/middleware/sidekiq_web_static.rb:20:in `call'","lib/gitlab/metrics/requests_rack_middleware.rb:75:in `call'","lib/gitlab/middleware/release_env.rb:13:in `call'"],"db_duration_s":0.04798,"view_duration_s":0.0,"duration_s":0.43985}
==> /var/log/gitlab/gitlab-rails/production.log <==
ActionView::Template::Error (14:failed to connect to all addresses.):
48: - if @project.empty_repo?
49: = render 'stat_anchor_list', anchors: @project.empty_repo_statistics_anchors
50: - else
51: = render 'stat_anchor_list', anchors: @project.statistics_anchors(show_auto_devops_callout: show_auto_devops_callout)
52:
53: .home-panel-home-desc.mt-1
54: - if @project.description.present?
lib/gitlab/git/wraps_gitaly_errors.rb:15:in `rescue in wrapped_gitaly_errors'
lib/gitlab/git/wraps_gitaly_errors.rb:6:in `wrapped_gitaly_errors'
lib/gitlab/git/repository.rb:148:in `branch_count'
app/models/repository.rb:569:in `branch_count'
lib/gitlab/repository_cache_adapter.rb:21:in `block (2 levels) in cache_method'
lib/gitlab/repository_cache.rb:25:in `fetch'
lib/gitlab/repository_cache_adapter.rb:163:in `block in cache_method_output'
lib/gitlab/utils/strong_memoize.rb:30:in `strong_memoize'
lib/gitlab/repository_cache_adapter.rb:203:in `block in memoize_method_output'
lib/gitlab/repository_cache_adapter.rb:212:in `no_repository_fallback'
lib/gitlab/repository_cache_adapter.rb:202:in `memoize_method_output'
lib/gitlab/repository_cache_adapter.rb:162:in `cache_method_output'
lib/gitlab/repository_cache_adapter.rb:20:in `block in cache_method'
app/presenters/project_presenter.rb:229:in `branches_anchor_data'
app/presenters/project_presenter.rb:30:in `statistics_anchors'
app/views/projects/_home_panel.html.haml:51
app/views/projects/_home_panel.html.haml:46
app/views/projects/show.html.haml:14
app/controllers/application_controller.rb:136:in `render'
app/controllers/application_controller.rb:555:in `block in allow_gitaly_ref_name_caching'
lib/gitlab/gitaly_client.rb:322:in `allow_ref_name_caching'
app/controllers/application_controller.rb:554:in `allow_gitaly_ref_name_caching'
app/controllers/application_controller.rb:504:in `set_current_admin'
lib/gitlab/session.rb:11:in `with_session'
app/controllers/application_controller.rb:495:in `set_session_storage'
lib/gitlab/i18n.rb:105:in `with_locale'
lib/gitlab/i18n.rb:111:in `with_user_locale'
app/controllers/application_controller.rb:489:in `set_locale'
app/controllers/application_controller.rb:483:in `set_current_context'
lib/gitlab/metrics/elasticsearch_rack_middleware.rb:16:in `call'
lib/gitlab/middleware/rails_queue_duration.rb:33:in `call'
lib/gitlab/middleware/speedscope.rb:13:in `call'
lib/gitlab/request_profiler/middleware.rb:17:in `call'
lib/gitlab/database/load_balancing/rack_middleware.rb:23:in `call'
lib/gitlab/metrics/rack_middleware.rb:16:in `block in call'
lib/gitlab/metrics/web_transaction.rb:46:in `run'
lib/gitlab/metrics/rack_middleware.rb:16:in `call'
lib/gitlab/jira/middleware.rb:19:in `call'
lib/gitlab/middleware/go.rb:20:in `call'
lib/gitlab/etag_caching/middleware.rb:21:in `call'
lib/gitlab/middleware/multipart.rb:173:in `call'
lib/gitlab/middleware/read_only/controller.rb:50:in `call'
lib/gitlab/middleware/read_only.rb:18:in `call'
lib/gitlab/middleware/same_site_cookies.rb:27:in `call'
lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'
lib/gitlab/middleware/basic_health_check.rb:25:in `call'
lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'
lib/gitlab/middleware/request_context.rb:21:in `call'
config/initializers/fix_local_cache_middleware.rb:11:in `call'
lib/gitlab/middleware/compressed_json.rb:26:in `call'
lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call'
lib/gitlab/middleware/sidekiq_web_static.rb:20:in `call'
lib/gitlab/metrics/requests_rack_middleware.rb:75:in `call'
lib/gitlab/middleware/release_env.rb:13:in `call'
Assumption
The Pi runs a 32 bit operating system and that seems to conflict with the structs of gitaly. There is an issue in the golang repo that describes the problem and offers a solution.
Thanks for helping
Edited by Ghost User