Execution of the command `/opt/gitlab/embedded/bin/gitlab-kas --version` failed with a non-zero exit code - GitLab 17.0 fails on CentOS 7
When updating GitLab from gitlab-ce-16.11.2-ce.0.el7.x86_64 to gitlab-runner-17.0.0-1.x86_64 on a CentOS 7 node, the update failed with the following error:
# cat /opt/gitlab/embedded/cookbooks/cache/cinc-stacktrace.out
Generated at 2024-05-16 14:39:37 -0400
RuntimeError: version_file[Create version file for Gitlab KAS] (gitlab-kas::enable line 67) had an error: RuntimeError: Execution of the command `/opt/gitlab/embedded/bin/gitlab-kas --version` failed with a non-zero exit code (2)
stdout:
stderr: panic: failed to parse "2024-05-15T09:24:36+0000" into RFC3339 compliant time object, because: parsing time "2024-05-15T09:24:36+0000" as "2006-01-02T15:04:05Z07:00": cannot parse "+0000" as "Z07:00". Fix the build process.
goroutine 1 [running]:
gitlab.com/gitlab-org/cluster-integration/gitlab-agent/v17/cmd.init.0()
/var/cache/omnibus/src/gitlab-kas/cmd/build_info.go:18 +0xf3
/opt/gitlab/embedded/cookbooks/cache/cookbooks/package/libraries/helpers/version_helper.rb:16:in `version'
/opt/gitlab/embedded/cookbooks/cache/cookbooks/package/resources/version_file.rb:32:in `block (2 levels) in class_from_file'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/resource_builder.rb:72:in `instance_exec'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/resource_builder.rb:72:in `build'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/dsl/declare_resource.rb:313:in `build_resource'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/dsl/declare_resource.rb:265:in `declare_resource'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/dsl/resources.rb:36:in `file'
/opt/gitlab/embedded/cookbooks/cache/cookbooks/package/resources/version_file.rb:31:in `block in class_from_file'
(eval):2:in `block in action_create'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/provider.rb:304:in `instance_eval'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/provider.rb:304:in `compile_and_converge_action'
(eval):2:in `action_create'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/provider.rb:245:in `run_action'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/resource.rb:601:in `block in run_action'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/resource.rb:628:in `with_umask'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/resource.rb:600:in `run_action'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/runner.rb:74:in `run_action'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/runner.rb:108:in `block in run_all_actions'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/runner.rb:108:in `each'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/runner.rb:108:in `run_all_actions'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/runner.rb:132:in `block in converge'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/resource_collection/resource_list.rb:96:in `block in execute_each_resource'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/resource_collection/stepable_iterator.rb:114:in `call_iterator_block'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/resource_collection/stepable_iterator.rb:85:in `step'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/resource_collection/stepable_iterator.rb:103:in `iterate'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/resource_collection/stepable_iterator.rb:54:in `each_with_index'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/resource_collection/resource_list.rb:94:in `execute_each_resource'
/opt/gitlab/embedded/lib/ruby/3.1.0/forwardable.rb:238:in `execute_each_resource'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/runner.rb:130:in `converge'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/client.rb:875:in `block in converge'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/client.rb:870:in `catch'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/client.rb:870:in `converge'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/client.rb:894:in `converge_and_save'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/client.rb:298:in `run'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/application.rb:305:in `run_with_graceful_exit_option'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/application.rb:281:in `block in run_chef_client'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/local_mode.rb:42:in `with_server_connectivity'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/application.rb:264:in `run_chef_client'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/application/base.rb:354:in `run_application'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/application.rb:67:in `run'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-bin-18.3.0/bin/cinc-client:25:in `<top (required)>'
/opt/gitlab/embedded/bin/cinc-client:25:in `load'
/opt/gitlab/embedded/bin/cinc-client:25:in `<main>'
>>>> Caused by RuntimeError: Execution of the command `/opt/gitlab/embedded/bin/gitlab-kas --version` failed with a non-zero exit code (2)
stdout:
stderr: panic: failed to parse "2024-05-15T09:24:36+0000" into RFC3339 compliant time object, because: parsing time "2024-05-15T09:24:36+0000" as "2006-01-02T15:04:05Z07:00": cannot parse "+0000" as "Z07:00". Fix the build process.
goroutine 1 [running]:
gitlab.com/gitlab-org/cluster-integration/gitlab-agent/v17/cmd.init.0()
/var/cache/omnibus/src/gitlab-kas/cmd/build_info.go:18 +0xf3
/opt/gitlab/embedded/cookbooks/cache/cookbooks/package/libraries/helpers/version_helper.rb:16:in `version'
/opt/gitlab/embedded/cookbooks/cache/cookbooks/package/resources/version_file.rb:32:in `block (2 levels) in class_from_file'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/resource_builder.rb:72:in `instance_exec'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/resource_builder.rb:72:in `build'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/dsl/declare_resource.rb:313:in `build_resource'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/dsl/declare_resource.rb:265:in `declare_resource'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/dsl/resources.rb:36:in `file'
/opt/gitlab/embedded/cookbooks/cache/cookbooks/package/resources/version_file.rb:31:in `block in class_from_file'
(eval):2:in `block in action_create'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/provider.rb:304:in `instance_eval'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/provider.rb:304:in `compile_and_converge_action'
(eval):2:in `action_create'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/provider.rb:245:in `run_action'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/resource.rb:601:in `block in run_action'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/resource.rb:628:in `with_umask'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/resource.rb:600:in `run_action'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/runner.rb:74:in `run_action'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/runner.rb:108:in `block in run_all_actions'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/runner.rb:108:in `each'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/runner.rb:108:in `run_all_actions'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/runner.rb:132:in `block in converge'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/resource_collection/resource_list.rb:96:in `block in execute_each_resource'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/resource_collection/stepable_iterator.rb:114:in `call_iterator_block'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/resource_collection/stepable_iterator.rb:85:in `step'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/resource_collection/stepable_iterator.rb:103:in `iterate'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/resource_collection/stepable_iterator.rb:54:in `each_with_index'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/resource_collection/resource_list.rb:94:in `execute_each_resource'
/opt/gitlab/embedded/lib/ruby/3.1.0/forwardable.rb:238:in `execute_each_resource'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/runner.rb:130:in `converge'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/client.rb:875:in `block in converge'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/client.rb:870:in `catch'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/client.rb:870:in `converge'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/client.rb:894:in `converge_and_save'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/client.rb:298:in `run'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/application.rb:305:in `run_with_graceful_exit_option'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/application.rb:281:in `block in run_chef_client'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/local_mode.rb:42:in `with_server_connectivity'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/application.rb:264:in `run_chef_client'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/application/base.rb:354:in `run_application'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/application.rb:67:in `run'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-bin-18.3.0/bin/cinc-client:25:in `<top (required)>'
/opt/gitlab/embedded/bin/cinc-client:25:in `load'
/opt/gitlab/embedded/bin/cinc-client:25:in `<main>'
The installed binary is indeed broken, and a clean environment does not help:
# env -i /opt/gitlab/embedded/bin/gitlab-kas --version
panic: failed to parse "2024-05-15T09:24:36+0000" into RFC3339 compliant time object, because: parsing time "2024-05-15T09:24:36+0000" as "2006-01-02T15:04:05Z07:00": cannot parse "+0000" as "Z07:00". Fix the build process.
goroutine 1 [running]:
gitlab.com/gitlab-org/cluster-integration/gitlab-agent/v17/cmd.init.0()
/var/cache/omnibus/src/gitlab-kas/cmd/build_info.go:18 +0xf3
Might be related to !1466 (merged)
Workaround
As a workaround, disabling the GitLab agent server for Kubernetes allow the upgrade to finish.
If gitlab-kas
is still needed, please use kas 16.11. More details in this comment
Edited by Emily Chang