gitlab-exporter throws Encoding::InvalidByteSequenceError after upgrading to GitLab v17.10.3
Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.
Summary
After upgrading our GitLab instance from v17.8.5 to v17.10.3 the gitlab-exporter started to throw exceptions every few minutes:
2025-04-03_21:35:34.25657 2025-04-03 23:35:34 - Encoding::InvalidByteSequenceError - "\xC3" on US-ASCII:
2025-04-03_21:35:34.25659 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/json-2.10.1/lib/json/common.rb:245:in `encode'
2025-04-03_21:35:34.25659 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/json-2.10.1/lib/json/common.rb:245:in `parse'
2025-04-03_21:35:34.25659 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/json-2.10.1/lib/json/common.rb:245:in `parse'
2025-04-03_21:35:34.25659 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/sidekiq-6.5.12/lib/sidekiq.rb:255:in `load_json'
2025-04-03_21:35:34.25659 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/sidekiq-6.5.12/lib/sidekiq/api.rb:1123:in `block (2 levels) in each'
2025-04-03_21:35:34.25660 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/sidekiq-6.5.12/lib/sidekiq/api.rb:1120:in `each_pair'
2025-04-03_21:35:34.25660 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/sidekiq-6.5.12/lib/sidekiq/api.rb:1120:in `block in each'
2025-04-03_21:35:34.25660 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/sidekiq-6.5.12/lib/sidekiq/api.rb:1119:in `each'
2025-04-03_21:35:34.25660 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/sidekiq-6.5.12/lib/sidekiq/api.rb:1119:in `each'
2025-04-03_21:35:34.25661 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/gitlab-exporter-15.2.0/lib/gitlab_exporter/sidekiq.rb:136:in `map'
2025-04-03_21:35:34.25662 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/gitlab-exporter-15.2.0/lib/gitlab_exporter/sidekiq.rb:136:in `block in probe_workers'
2025-04-03_21:35:34.25662 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/gitlab-exporter-15.2.0/lib/gitlab_exporter/sidekiq.rb:191:in `block in with_sidekiq'
2025-04-03_21:35:34.25662 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/gitlab-exporter-15.2.0/lib/gitlab_exporter/sidekiq.rb:184:in `synchronize'
2025-04-03_21:35:34.25662 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/gitlab-exporter-15.2.0/lib/gitlab_exporter/sidekiq.rb:184:in `with_sidekiq'
2025-04-03_21:35:34.25662 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/gitlab-exporter-15.2.0/lib/gitlab_exporter/sidekiq.rb:133:in `probe_workers'
2025-04-03_21:35:34.25662 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/gitlab-exporter-15.2.0/lib/gitlab_exporter/prober.rb:18:in `block (3 levels) in probe_all'
2025-04-03_21:35:34.25662 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/gitlab-exporter-15.2.0/lib/gitlab_exporter/prober.rb:17:in `each'
2025-04-03_21:35:34.25662 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/gitlab-exporter-15.2.0/lib/gitlab_exporter/prober.rb:17:in `block (2 levels) in probe_all'
2025-04-03_21:35:34.25663 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/gitlab-exporter-15.2.0/lib/gitlab_exporter/prober.rb:15:in `each'
2025-04-03_21:35:34.25663 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/gitlab-exporter-15.2.0/lib/gitlab_exporter/prober.rb:15:in `block in probe_all'
2025-04-03_21:35:34.25663 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/gitlab-exporter-15.2.0/lib/gitlab_exporter/prober.rb:14:in `each'
2025-04-03_21:35:34.25663 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/gitlab-exporter-15.2.0/lib/gitlab_exporter/prober.rb:14:in `probe_all'
2025-04-03_21:35:34.25663 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/gitlab-exporter-15.2.0/lib/gitlab_exporter/web_exporter.rb:139:in `block (2 levels) in setup_probes'
2025-04-03_21:35:34.25663 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/sinatra-2.2.4/lib/sinatra/base.rb:1697:in `call'
2025-04-03_21:35:34.25663 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/sinatra-2.2.4/lib/sinatra/base.rb:1697:in `block in compile!'
2025-04-03_21:35:34.25664 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/sinatra-2.2.4/lib/sinatra/base.rb:1030:in `block (3 levels) in route!'
2025-04-03_21:35:34.25664 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/sinatra-2.2.4/lib/sinatra/base.rb:1049:in `route_eval'
2025-04-03_21:35:34.25664 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/sinatra-2.2.4/lib/sinatra/base.rb:1030:in `block (2 levels) in route!'
2025-04-03_21:35:34.25664 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/sinatra-2.2.4/lib/sinatra/base.rb:1078:in `block in process_route'
2025-04-03_21:35:34.25664 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/sinatra-2.2.4/lib/sinatra/base.rb:1076:in `catch'
2025-04-03_21:35:34.25664 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/sinatra-2.2.4/lib/sinatra/base.rb:1076:in `process_route'
2025-04-03_21:35:34.25664 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/sinatra-2.2.4/lib/sinatra/base.rb:1028:in `block in route!'
2025-04-03_21:35:34.25665 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/sinatra-2.2.4/lib/sinatra/base.rb:1025:in `each'
2025-04-03_21:35:34.25665 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/sinatra-2.2.4/lib/sinatra/base.rb:1025:in `route!'
2025-04-03_21:35:34.25665 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/sinatra-2.2.4/lib/sinatra/base.rb:1147:in `block in dispatch!'
2025-04-03_21:35:34.25665 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/sinatra-2.2.4/lib/sinatra/base.rb:1119:in `block in invoke'
2025-04-03_21:35:34.25665 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/sinatra-2.2.4/lib/sinatra/base.rb:1119:in `catch'
2025-04-03_21:35:34.25665 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/sinatra-2.2.4/lib/sinatra/base.rb:1119:in `invoke'
2025-04-03_21:35:34.25666 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/sinatra-2.2.4/lib/sinatra/base.rb:1142:in `dispatch!'
2025-04-03_21:35:34.25666 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/sinatra-2.2.4/lib/sinatra/base.rb:956:in `block in call!'
2025-04-03_21:35:34.25666 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/sinatra-2.2.4/lib/sinatra/base.rb:1119:in `block in invoke'
2025-04-03_21:35:34.25666 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/sinatra-2.2.4/lib/sinatra/base.rb:1119:in `catch'
2025-04-03_21:35:34.25666 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/sinatra-2.2.4/lib/sinatra/base.rb:1119:in `invoke'
2025-04-03_21:35:34.25666 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/sinatra-2.2.4/lib/sinatra/base.rb:956:in `call!'
2025-04-03_21:35:34.25666 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/sinatra-2.2.4/lib/sinatra/base.rb:945:in `call'
2025-04-03_21:35:34.25666 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/gitlab-exporter-15.2.0/lib/gitlab_exporter/web_exporter.rb:51:in `call'
2025-04-03_21:35:34.25667 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/rack-2.2.11/lib/rack/logger.rb:17:in `call'
2025-04-03_21:35:34.25667 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/gitlab-exporter-15.2.0/lib/gitlab_exporter/web_exporter.rb:25:in `call'
2025-04-03_21:35:34.25667 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/rack-protection-2.2.4/lib/rack/protection/xss_header.rb:18:in `call'
2025-04-03_21:35:34.25667 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/rack-protection-2.2.4/lib/rack/protection/path_traversal.rb:16:in `call'
2025-04-03_21:35:34.25667 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/rack-protection-2.2.4/lib/rack/protection/json_csrf.rb:26:in `call'
2025-04-03_21:35:34.25667 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/rack-protection-2.2.4/lib/rack/protection/base.rb:50:in `call'
2025-04-03_21:35:34.25667 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/rack-protection-2.2.4/lib/rack/protection/base.rb:50:in `call'
2025-04-03_21:35:34.25668 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/rack-protection-2.2.4/lib/rack/protection/frame_options.rb:31:in `call'
2025-04-03_21:35:34.25668 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/rack-2.2.11/lib/rack/null_logger.rb:11:in `call'
2025-04-03_21:35:34.25668 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/rack-2.2.11/lib/rack/head.rb:12:in `call'
2025-04-03_21:35:34.25668 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/sinatra-2.2.4/lib/sinatra/show_exceptions.rb:22:in `call'
2025-04-03_21:35:34.25668 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/sinatra-2.2.4/lib/sinatra/base.rb:218:in `call'
2025-04-03_21:35:34.25668 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/sinatra-2.2.4/lib/sinatra/base.rb:2004:in `call'
2025-04-03_21:35:34.25668 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/sinatra-2.2.4/lib/sinatra/base.rb:1564:in `block in call'
2025-04-03_21:35:34.25668 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/sinatra-2.2.4/lib/sinatra/base.rb:1780:in `synchronize'
2025-04-03_21:35:34.25669 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/sinatra-2.2.4/lib/sinatra/base.rb:1564:in `call'
2025-04-03_21:35:34.25669 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/rack-2.2.11/lib/rack/handler/webrick.rb:95:in `service'
2025-04-03_21:35:34.25669 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/webrick-1.8.2/lib/webrick/httpserver.rb:140:in `service'
2025-04-03_21:35:34.25669 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/webrick-1.8.2/lib/webrick/httpserver.rb:96:in `run'
2025-04-03_21:35:34.25669 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/webrick-1.8.2/lib/webrick/server.rb:310:in `block in start_thread'
2025-04-03_21:35:34.27172 10.0.1.37 - - [03/Apr/2025:23:35:34 CEST] "GET /sidekiq HTTP/1.1" 500 184261In #531486 (moved) and omnibus-gitlab#8995 (closed) a similar error message during upgrades was reported, and a new json gem version was suspected to be the root cause.
This new version 2.10.1 is also being called here in the exporter:
2025-04-03_21:35:34.25659 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/json-2.10.1/lib/json/common.rb:245:in `parse'
2025-04-03_21:35:34.25659 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/sidekiq-6.5.12/lib/sidekiq.rb:255:in `load_json'What is the current bug behavior?
The gitlab-exporter throws this error every 2-3 minutes, resulting in missing metrics in Prometheus:
$ cat /var/log/gitlab/gitlab-exporter | grep "InvalidByteSequence"
2025-04-04_05:50:49.25334 2025-04-04 07:50:49 - Encoding::InvalidByteSequenceError - "\xC3" on US-ASCII:
2025-04-04_05:54:34.25427 2025-04-04 07:54:34 - Encoding::InvalidByteSequenceError - "\xC3" on US-ASCII:
2025-04-04_05:55:19.25927 2025-04-04 07:55:19 - Encoding::InvalidByteSequenceError - "\xC3" on US-ASCII:
2025-04-04_06:05:34.26005 2025-04-04 08:05:34 - Encoding::InvalidByteSequenceError - "\xC3" on US-ASCII:
2025-04-04_06:05:49.26584 2025-04-04 08:05:49 - Encoding::InvalidByteSequenceError - "\xC3" on US-ASCII:
2025-04-04_06:08:49.26050 2025-04-04 08:08:49 - Encoding::InvalidByteSequenceError - "\xC3" on US-ASCII:
2025-04-04_06:10:34.25794 2025-04-04 08:10:34 - Encoding::InvalidByteSequenceError - "\xC3" on US-ASCII:
2025-04-04_06:10:49.25630 2025-04-04 08:10:49 - Encoding::InvalidByteSequenceError - "\xC3" on US-ASCII:
2025-04-04_06:15:19.26265 2025-04-04 08:15:19 - Encoding::InvalidByteSequenceError - "\xC3" on US-ASCII:
2025-04-04_06:15:34.26514 2025-04-04 08:15:34 - Encoding::InvalidByteSequenceError - "\xC3" on US-ASCII:
2025-04-04_06:20:34.26435 2025-04-04 08:20:34 - Encoding::InvalidByteSequenceError - "\xC3" on US-ASCII:
2025-04-04_06:21:19.25600 2025-04-04 08:21:19 - Encoding::InvalidByteSequenceError - "\xC3" on US-ASCII:
2025-04-04_06:24:19.25468 2025-04-04 08:24:19 - Encoding::InvalidByteSequenceError - "\xC3" on US-ASCII:
2025-04-04_06:25:19.25978 2025-04-04 08:25:19 - Encoding::InvalidByteSequenceError - "\xC3" on US-ASCII:
2025-04-04_06:25:34.25856 2025-04-04 08:25:34 - Encoding::InvalidByteSequenceError - "\xC3" on US-ASCII:
2025-04-04_06:25:49.25758 2025-04-04 08:25:49 - Encoding::InvalidByteSequenceError - "\xC3" on US-ASCII:
2025-04-04_06:26:19.25419 2025-04-04 08:26:19 - Encoding::InvalidByteSequenceError - "\xC3" on US-ASCII:
2025-04-04_06:29:19.26866 2025-04-04 08:29:19 - Encoding::InvalidByteSequenceError - "\xC3" on US-ASCII:
2025-04-04_06:30:19.26287 2025-04-04 08:30:19 - Encoding::InvalidByteSequenceError - "\xC3" on US-ASCII:
2025-04-04_06:33:04.25321 2025-04-04 08:33:04 - Encoding::InvalidByteSequenceError - "\xC3" on US-ASCII:
2025-04-04_06:35:19.25992 2025-04-04 08:35:19 - Encoding::InvalidByteSequenceError - "\xC3" on US-ASCII:
2025-04-04_06:35:34.28043 2025-04-04 08:35:34 - Encoding::InvalidByteSequenceError - "\xC3" on US-ASCII:
2025-04-04_06:38:34.25362 2025-04-04 08:38:34 - Encoding::InvalidByteSequenceError - "\xC3" on US-ASCII:Because we are actively monitoring some of these metrics, we are getting many NoData alerts from Grafana because the data points are missing.
What is the expected correct behavior?
The gitlab-exporter should not throw this error and instead serve the metrics, as it was the case in GitLab v17.8.5.
Results of GitLab environment info
Expand for output related to GitLab environment info
System information System: Debian 11 Current User: git Using RVM: no Ruby Version: 3.2.5 Gem Version: 3.6.5 Bundler Version:2.6.5 Rake Version: 13.0.6 Redis Version: 7.0.15 Sidekiq Version:7.2.4 Go Version: unknown GitLab information Version: 17.10.3 Revision: 22d4014a923 Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: PostgreSQL DB Version: 14.17 URL: https://git.company.local HTTP Clone URL: https://git.company.local/some-group/some-project.git SSH Clone URL: git@git.company.local:some-group/some-project.git Using LDAP: yes Using Omniauth: yes Omniauth Providers: saml GitLab Shell Version: 14.41.0 Repository storages: - default: unix:/var/opt/gitlab/gitaly/gitaly.socket GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell Gitaly - default Address: unix:/var/opt/gitlab/gitaly/gitaly.socket - default Version: 17.10.3 - default Git Version: 2.48.1.gl1
Possible fixes
I assume the upgraded json gem is the root cause for this, maybe a downgrade would be useful?
Edited by 🤖 GitLab Bot 🤖