x509 failures on gitlab updates

Summary

During GitLab updates, the x509 signature update task fails with GRPC deadline exceeded errors.

This failure does not happen all the time. Therefore, it seems to be timing related error. Maybe too much load on the first run. When executing it a 2nd time, the rake task works because the data is probably cached.

Steps to reproduce

Run GitLab update process
The gitlab-rake gitlab:x509:update_signatures command runs
After a while, the task fails with the error rake aborted! GRPC::DeadlineExceeded: 4:Deadline Exceeded. debug_error_string:{UNKNOWN:Error received from peer {grpc_message:\"Deadline Exceeded\", grpc_status:4, created_time:"2025-05-24T08:34:18.000533088+00:00"}}

Proposal / Checklist

Add automatic retry mechanism in case a X509CommitSignature has failed => Update x509 signatures: Add automatic retry on ... (!192543 - merged)
Improve error handling and / or logging to identify problematic CommitSignatures::X509CommitSignature records that are updated in update.rake#L23 => chore(rake): Add logging for x509 commit signat... (!192504 - merged)

Example Project

What is the current bug behavior?

The x509 signature update task fails with a GRPC deadline exceeded error.

What is the expected correct behavior?

The x509 signature update task should complete successfully without timeout errors or have a better error handling, e.g. automatic retry.

Relevant logs and/or screenshots

The error occurs in the following call chain:

/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client.rb:292:in 'execute'
Leading to /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client/repository_service.rb:22:in 'exists?'
Eventually reaching /opt/gitlab/embedded/service/gitlab-rails/app/models/commit_signatures/x509_commit_signature.rb:16:in 'x509_commit'
And failing in /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/x509/update.rake:23:in 'block in update_certificates'

Click to expand the console output with the error message

TASK [../roles/code-root-ca : Root CA | Update all X509 signatures] ************************************************************************************************************************************************************************
Saturday 24 May 2025  08:33:05 +0000 (0:00:30.251)       0:04:26.926 **********
skipping: [frontend-01.code.example.com]
skipping: [frontend-02.code.example.com]
skipping: [frontend-03.code.example.com]
skipping: [frontend-04.code.example.com]
skipping: [frontend-05.code.example.com]
skipping: [frontend-06.code.example.com]
skipping: [frontend-07.code.example.com]
skipping: [frontend-08.code.example.com]
skipping: [frontend-09.code.example.com]
fatal: [frontend-primary.code.example.com]: FAILED! => {"changed": true, "cmd": ["gitlab-rake", "gitlab:x509:update_signatures"], "delta": "0:01:12.026203", "end": "2025-05-24 08:34:18.219674", "msg": "non-zero return code", "rc": 1,
"start": "2025-05-24 08:33:06.193471", "stderr": "rake aborted!\nGRPC::DeadlineExceeded: 4:Deadline Exceeded. debug_error_string:{UNKNOWN:Error received from peer  {grpc_message:\"Deadline Exceeded\", grpc_status:4, created_time:\"2025-
05-24T08:34:18.000533088+00:00\"}}\n/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client.rb:292:in `execute'\n/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client/call.rb:19:in `block in call'\n/opt/gitlab/em
bedded/service/gitlab-rails/lib/gitlab/gitaly_client/call.rb:61:in `recording_request'\n/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client/call.rb:18:in `call'\n/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly
_client.rb:281:in `call'\n/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client/with_feature_flag_actors.rb:31:in `block in gitaly_client_call'\n/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client.rb:650:in `
with_feature_flag_actors'\n/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client/with_feature_flag_actors.rb:25:in `gitaly_client_call'\n/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client/repository_service.
rb:22:in `exists?'\n/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/git/repository.rb:97:in `exists?'\n/opt/gitlab/embedded/service/gitlab-rails/app/models/repository.rb:576:in `exists?'\n/opt/gitlab/embedded/service/gitlab-rails/l
ib/gitlab/repository_cache_adapter.rb:95:in `block (2 levels) in cache_method_asymmetrically'\n/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/repository_cache.rb:44:in `fetch_without_caching_false'\n/opt/gitlab/embedded/service/gi
tlab-rails/lib/gitlab/repository_cache_adapter.rb:190:in `block (2 levels) in cache_method_output_asymmetrically'\n/opt/gitlab/embedded/service/gitlab-rails/gems/gitlab-safe_request_store/lib/gitlab/safe_request_store/null_store.rb:37:i
n `fetch'\n/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/repository_cache.rb:25:in `fetch'\n/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/repository_cache_adapter.rb:189:in `block in cache_method_output_asymmetrically'\n/o
pt/gitlab/embedded/service/gitlab-rails/gems/gitlab-utils/lib/gitlab/utils/strong_memoize.rb:34:in `strong_memoize'\n/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/repository_cache_adapter.rb:203:in `block in memoize_method_output
'\n/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/repository_cache_adapter.rb:212:in `no_repository_fallback'\n/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/repository_cache_adapter.rb:202:in `memoize_method_output'\n/opt/g
itlab/embedded/service/gitlab-rails/lib/gitlab/repository_cache_adapter.rb:188:in `cache_method_output_asymmetrically'\n/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/repository_cache_adapter.rb:94:in `block in cache_method_asymme
trically'\n/opt/gitlab/embedded/service/gitlab-rails/app/models/repository.rb:123:in `commit'\n/opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/has_repository.rb:43:in `commit'\n/opt/gitlab/embedded/service/gitlab-rails/app
/models/concerns/commit_signature.rb:38:in `commit'\n/opt/gitlab/embedded/service/gitlab-rails/app/models/commit_signatures/x509_commit_signature.rb:16:in `x509_commit'\n/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/x509/up
date.rake:23:in `block in update_certificates'\n/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/x509/update.rake:22:in `update_certificates'\n/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/x509/update.rake:9:in `b
lock (3 levels) in <top (required)>'\n/opt/gitlab/embedded/bin/bundle:25:in `load'\n/opt/gitlab/embedded/bin/bundle:25:in `<main>'\nTasks: TOP => gitlab:x509:update_signatures\n(See full trace by running task with --trace)", "stderr_lin
es": ["rake aborted!", "GRPC::DeadlineExceeded: 4:Deadline Exceeded. debug_error_string:{UNKNOWN:Error received from peer  {grpc_message:\"Deadline Exceeded\", grpc_status:4, created_time:\"2025-05-24T08:34:18.000533088+00:00\"}}", "/op
t/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client.rb:292:in `execute'", "/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client/call.rb:19:in `block in call'", "/opt/gitlab/embedded/service/gitlab-rails/lib/git
lab/gitaly_client/call.rb:61:in `recording_request'", "/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client/call.rb:18:in `call'", "/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client.rb:281:in `call'", "/op
t/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client/with_feature_flag_actors.rb:31:in `block in gitaly_client_call'", "/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client.rb:650:in `with_feature_flag_actors'",
 "/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client/with_feature_flag_actors.rb:25:in `gitaly_client_call'", "/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client/repository_service.rb:22:in `exists?'", "/
opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/git/repository.rb:97:in `exists?'", "/opt/gitlab/embedded/service/gitlab-rails/app/models/repository.rb:576:in `exists?'", "/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/reposit
ory_cache_adapter.rb:95:in `block (2 levels) in cache_method_asymmetrically'", "/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/repository_cache.rb:44:in `fetch_without_caching_false'", "/opt/gitlab/embedded/service/gitlab-rails/li
b/gitlab/repository_cache_adapter.rb:190:in `block (2 levels) in cache_method_output_asymmetrically'", "/opt/gitlab/embedded/service/gitlab-rails/gems/gitlab-safe_request_store/lib/gitlab/safe_request_store/null_store.rb:37:in `fetch'",
 "/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/repository_cache.rb:25:in `fetch'", "/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/repository_cache_adapter.rb:189:in `block in cache_method_output_asymmetrically'", "/opt/gi
tlab/embedded/service/gitlab-rails/gems/gitlab-utils/lib/gitlab/utils/strong_memoize.rb:34:in `strong_memoize'", "/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/repository_cache_adapter.rb:203:in `block in memoize_method_output'",
 "/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/repository_cache_adapter.rb:212:in `no_repository_fallback'", "/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/repository_cache_adapter.rb:202:in `memoize_method_output'", "/op
t/gitlab/embedded/service/gitlab-rails/lib/gitlab/repository_cache_adapter.rb:188:in `cache_method_output_asymmetrically'", "/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/repository_cache_adapter.rb:94:in `block in cache_method_a
symmetrically'", "/opt/gitlab/embedded/service/gitlab-rails/app/models/repository.rb:123:in `commit'", "/opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/has_repository.rb:43:in `commit'", "/opt/gitlab/embedded/service/gitla
b-rails/app/models/concerns/commit_signature.rb:38:in `commit'", "/opt/gitlab/embedded/service/gitlab-rails/app/models/commit_signatures/x509_commit_signature.rb:16:in `x509_commit'", "/opt/gitlab/embedded/service/gitlab-rails/lib/tasks
/gitlab/x509/update.rake:23:in `block in update_certificates'", "/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/x509/update.rake:22:in `update_certificates'", "/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/x509/
update.rake:9:in `block (3 levels) in <top (required)>'", "/opt/gitlab/embedded/bin/bundle:25:in `load'", "/opt/gitlab/embedded/bin/bundle:25:in `<main>'", "Tasks: TOP => gitlab:x509:update_signatures", "(See full trace by running task
with --trace)"], "stdout": "I, [2025-05-24T08:33:32.528996 #4024811]  INFO -- : Start to update x509 commit signatures", "stdout_lines": ["I, [2025-05-24T08:33:32.528996 #4024811]  INFO -- : Start to update x509 commit signatures"]}

Click to expand the pretty formated console output

fatal: [frontend-primary.code.siemens.rocks]: FAILED! => {
    "changed": true,
    "cmd": ["gitlab-rake", "gitlab:x509:update_signatures"],
    "delta": "0:01:12.026203",
    "end": "2025-05-24 08:34:18.219674",
    "msg": "non-zero return code",
    "rc": 1,
    "start": "2025-05-24 08:33:06.193471",
    "stderr": "rake aborted!\nGRPC::DeadlineExceeded: 4:Deadline Exceeded. debug_error_string:{UNKNOWN:Error received from peer  {grpc_message:\"Deadline Exceeded\", grpc_status:4, created_time:\"2025-05-24T08:34:18.000533088+00:00\"}}\n/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client.rb:292:in `execute'\n/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client/call.rb:19:in `block in call'\n/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client/call.rb:61:in `recording_request'\n/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client/call.rb:18:in `call'\n/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client.rb:281:in `call'\n/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client/with_feature_flag_actors.rb:31:in `block in gitaly_client_call'\n/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client.rb:650:in `with_feature_flag_actors'\n/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client/with_feature_flag_actors.rb:25:in `gitaly_client_call'\n/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client/repository_service.rb:22:in `exists?'\n/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/git/repository.rb:97:in `exists?'\n/opt/gitlab/embedded/service/gitlab-rails/app/models/repository.rb:576:in `exists?'\n/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/repository_cache_adapter.rb:95:in `block (2 levels) in cache_method_asymmetrically'\n/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/repository_cache.rb:44:in `fetch_without_caching_false'\n/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/repository_cache_adapter.rb:190:in `block (2 levels) in cache_method_output_asymmetrically'\n/opt/gitlab/embedded/service/gitlab-rails/gems/gitlab-safe_request_store/lib/gitlab/safe_request_store/null_store.rb:37:in `fetch'\n/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/repository_cache.rb:25:in `fetch'\n/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/repository_cache_adapter.rb:189:in `block in cache_method_output_asymmetrically'\n/opt/gitlab/embedded/service/gitlab-rails/gems/gitlab-utils/lib/gitlab/utils/strong_memoize.rb:34:in `strong_memoize'\n/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/repository_cache_adapter.rb:203:in `block in memoize_method_output'\n/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/repository_cache_adapter.rb:212:in `no_repository_fallback'\n/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/repository_cache_adapter.rb:202:in `memoize_method_output'\n/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/repository_cache_adapter.rb:188:in `cache_method_output_asymmetrically'\n/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/repository_cache_adapter.rb:94:in `block in cache_method_asymmetrically'\n/opt/gitlab/embedded/service/gitlab-rails/app/models/repository.rb:123:in `commit'\n/opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/has_repository.rb:43:in `commit'\n/opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/commit_signature.rb:38:in `commit'\n/opt/gitlab/embedded/service/gitlab-rails/app/models/commit_signatures/x509_commit_signature.rb:16:in `x509_commit'\n/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/x509/update.rake:23:in `block in update_certificates'\n/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/x509/update.rake:22:in `update_certificates'\n/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/x509/update.rake:9:in `block (3 levels) in <top (required)>'\n/opt/gitlab/embedded/bin/bundle:25:in `load'\n/opt/gitlab/embedded/bin/bundle:25:in `<main>'\nTasks: TOP => gitlab:x509:update_signatures\n(See full trace by running task with --trace)",
    "stderr_lines": [
        "rake aborted!",
        "GRPC::DeadlineExceeded: 4:Deadline Exceeded. debug_error_string:{UNKNOWN:Error received from peer  {grpc_message:\"Deadline Exceeded\", grpc_status:4, created_time:\"2025-05-24T08:34:18.000533088+00:00\"}}",
        "/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client.rb:292:in `execute'",
        "/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client/call.rb:19:in `block in call'",
        "/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client/call.rb:61:in `recording_request'",
        "/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client/call.rb:18:in `call'",
        "/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client.rb:281:in `call'",
        "/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client/with_feature_flag_actors.rb:31:in `block in gitaly_client_call'",
        "/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client.rb:650:in `with_feature_flag_actors'",
        "/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client/with_feature_flag_actors.rb:25:in `gitaly_client_call'",
        "/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client/repository_service.rb:22:in `exists?'",
        "/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/git/repository.rb:97:in `exists?'",
        "/opt/gitlab/embedded/service/gitlab-rails/app/models/repository.rb:576:in `exists?'",
        "/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/repository_cache_adapter.rb:95:in `block (2 levels) in cache_method_asymmetrically'",
        "/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/repository_cache.rb:44:in `fetch_without_caching_false'",
        "/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/repository_cache_adapter.rb:190:in `block (2 levels) in cache_method_output_asymmetrically'",
        "/opt/gitlab/embedded/service/gitlab-rails/gems/gitlab-safe_request_store/lib/gitlab/safe_request_store/null_store.rb:37:in `fetch'",
        "/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/repository_cache.rb:25:in `fetch'",
        "/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/repository_cache_adapter.rb:189:in `block in cache_method_output_asymmetrically'",
        "/opt/gitlab/embedded/service/gitlab-rails/gems/gitlab-utils/lib/gitlab/utils/strong_memoize.rb:34:in `strong_memoize'",
        "/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/repository_cache_adapter.rb:203:in `block in memoize_method_output'",
        "/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/repository_cache_adapter.rb:212:in `no_repository_fallback'",
        "/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/repository_cache_adapter.rb:202:in `memoize_method_output'",
        "/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/repository_cache_adapter.rb:188:in `cache_method_output_asymmetrically'",
        "/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/repository_cache_adapter.rb:94:in `block in cache_method_asymmetrically'",
        "/opt/gitlab/embedded/service/gitlab-rails/app/models/repository.rb:123:in `commit'",
        "/opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/has_repository.rb:43:in `commit'",
        "/opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/commit_signature.rb:38:in `commit'",
        "/opt/gitlab/embedded/service/gitlab-rails/app/models/commit_signatures/x509_commit_signature.rb:16:in `x509_commit'",
        "/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/x509/update.rake:23:in `block in update_certificates'",
        "/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/x509/update.rake:22:in `update_certificates'",
        "/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/x509/update.rake:9:in `block (3 levels) in <top (required)>'",
        "/opt/gitlab/embedded/bin/bundle:25:in `load'",
        "/opt/gitlab/embedded/bin/bundle:25:in `<main>'",
        "Tasks: TOP => gitlab:x509:update_signatures",
        "(See full trace by running task with --trace)"
    ],
    "stdout": "I, [2025-05-24T08:33:32.528996 #4024811]  INFO -- : Start to update x509 commit signatures",
    "stdout_lines": [
        "I, [2025-05-24T08:33:32.528996 #4024811]  INFO -- : Start to update x509 commit signatures"
    ]
}

Output of checks

Results of GitLab environment info

Expand for output related to GitLab environment info

Results of GitLab application Check

Expand for output related to the GitLab application check

Edited Jun 05, 2025 by Gerardo Navarro