gitlab-exporter throws Encoding::InvalidByteSequenceError after upgrading to GitLab v17.10.3 (#534220) · Issues · GitLab.org / GitLab

gitlab-exporter throws Encoding::InvalidByteSequenceError after upgrading to GitLab v17.10.3

<details> <summary> Everyone can contribute. [Help move this issue forward](https://handbook.gitlab.com/handbook/marketing/developer-relations/contributor-success/community-contributors-workflows/#contributor-links) while earning points, leveling up and collecting rewards. </summary> - [Close this issue](https://contributors.gitlab.com/manage-issue?action=close&projectId=278964&issueIid=534220) </details>  ### Summary After upgrading our GitLab instance from v17.8.5 to v17.10.3 the `gitlab-exporter` started to throw exceptions every few minutes: ``` 2025-04-03_21:35:34.25657 2025-04-03 23:35:34 - Encoding::InvalidByteSequenceError - "\xC3" on US-ASCII: 2025-04-03_21:35:34.25659 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/json-2.10.1/lib/json/common.rb:245:in `encode' 2025-04-03_21:35:34.25659 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/json-2.10.1/lib/json/common.rb:245:in `parse' 2025-04-03_21:35:34.25659 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/json-2.10.1/lib/json/common.rb:245:in `parse' 2025-04-03_21:35:34.25659 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/sidekiq-6.5.12/lib/sidekiq.rb:255:in `load_json' 2025-04-03_21:35:34.25659 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/sidekiq-6.5.12/lib/sidekiq/api.rb:1123:in `block (2 levels) in each' 2025-04-03_21:35:34.25660 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/sidekiq-6.5.12/lib/sidekiq/api.rb:1120:in `each_pair' 2025-04-03_21:35:34.25660 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/sidekiq-6.5.12/lib/sidekiq/api.rb:1120:in `block in each' 2025-04-03_21:35:34.25660 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/sidekiq-6.5.12/lib/sidekiq/api.rb:1119:in `each' 2025-04-03_21:35:34.25660 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/sidekiq-6.5.12/lib/sidekiq/api.rb:1119:in `each' 2025-04-03_21:35:34.25661 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/gitlab-exporter-15.2.0/lib/gitlab_exporter/sidekiq.rb:136:in `map' 2025-04-03_21:35:34.25662 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/gitlab-exporter-15.2.0/lib/gitlab_exporter/sidekiq.rb:136:in `block in probe_workers' 2025-04-03_21:35:34.25662 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/gitlab-exporter-15.2.0/lib/gitlab_exporter/sidekiq.rb:191:in `block in with_sidekiq' 2025-04-03_21:35:34.25662 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/gitlab-exporter-15.2.0/lib/gitlab_exporter/sidekiq.rb:184:in `synchronize' 2025-04-03_21:35:34.25662 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/gitlab-exporter-15.2.0/lib/gitlab_exporter/sidekiq.rb:184:in `with_sidekiq' 2025-04-03_21:35:34.25662 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/gitlab-exporter-15.2.0/lib/gitlab_exporter/sidekiq.rb:133:in `probe_workers' 2025-04-03_21:35:34.25662 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/gitlab-exporter-15.2.0/lib/gitlab_exporter/prober.rb:18:in `block (3 levels) in probe_all' 2025-04-03_21:35:34.25662 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/gitlab-exporter-15.2.0/lib/gitlab_exporter/prober.rb:17:in `each' 2025-04-03_21:35:34.25662 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/gitlab-exporter-15.2.0/lib/gitlab_exporter/prober.rb:17:in `block (2 levels) in probe_all' 2025-04-03_21:35:34.25663 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/gitlab-exporter-15.2.0/lib/gitlab_exporter/prober.rb:15:in `each' 2025-04-03_21:35:34.25663 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/gitlab-exporter-15.2.0/lib/gitlab_exporter/prober.rb:15:in `block in probe_all' 2025-04-03_21:35:34.25663 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/gitlab-exporter-15.2.0/lib/gitlab_exporter/prober.rb:14:in `each' 2025-04-03_21:35:34.25663 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/gitlab-exporter-15.2.0/lib/gitlab_exporter/prober.rb:14:in `probe_all' 2025-04-03_21:35:34.25663 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/gitlab-exporter-15.2.0/lib/gitlab_exporter/web_exporter.rb:139:in `block (2 levels) in setup_probes' 2025-04-03_21:35:34.25663 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/sinatra-2.2.4/lib/sinatra/base.rb:1697:in `call' 2025-04-03_21:35:34.25663 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/sinatra-2.2.4/lib/sinatra/base.rb:1697:in `block in compile!' 2025-04-03_21:35:34.25664 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/sinatra-2.2.4/lib/sinatra/base.rb:1030:in `block (3 levels) in route!' 2025-04-03_21:35:34.25664 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/sinatra-2.2.4/lib/sinatra/base.rb:1049:in `route_eval' 2025-04-03_21:35:34.25664 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/sinatra-2.2.4/lib/sinatra/base.rb:1030:in `block (2 levels) in route!' 2025-04-03_21:35:34.25664 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/sinatra-2.2.4/lib/sinatra/base.rb:1078:in `block in process_route' 2025-04-03_21:35:34.25664 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/sinatra-2.2.4/lib/sinatra/base.rb:1076:in `catch' 2025-04-03_21:35:34.25664 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/sinatra-2.2.4/lib/sinatra/base.rb:1076:in `process_route' 2025-04-03_21:35:34.25664 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/sinatra-2.2.4/lib/sinatra/base.rb:1028:in `block in route!' 2025-04-03_21:35:34.25665 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/sinatra-2.2.4/lib/sinatra/base.rb:1025:in `each' 2025-04-03_21:35:34.25665 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/sinatra-2.2.4/lib/sinatra/base.rb:1025:in `route!' 2025-04-03_21:35:34.25665 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/sinatra-2.2.4/lib/sinatra/base.rb:1147:in `block in dispatch!' 2025-04-03_21:35:34.25665 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/sinatra-2.2.4/lib/sinatra/base.rb:1119:in `block in invoke' 2025-04-03_21:35:34.25665 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/sinatra-2.2.4/lib/sinatra/base.rb:1119:in `catch' 2025-04-03_21:35:34.25665 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/sinatra-2.2.4/lib/sinatra/base.rb:1119:in `invoke' 2025-04-03_21:35:34.25666 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/sinatra-2.2.4/lib/sinatra/base.rb:1142:in `dispatch!' 2025-04-03_21:35:34.25666 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/sinatra-2.2.4/lib/sinatra/base.rb:956:in `block in call!' 2025-04-03_21:35:34.25666 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/sinatra-2.2.4/lib/sinatra/base.rb:1119:in `block in invoke' 2025-04-03_21:35:34.25666 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/sinatra-2.2.4/lib/sinatra/base.rb:1119:in `catch' 2025-04-03_21:35:34.25666 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/sinatra-2.2.4/lib/sinatra/base.rb:1119:in `invoke' 2025-04-03_21:35:34.25666 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/sinatra-2.2.4/lib/sinatra/base.rb:956:in `call!' 2025-04-03_21:35:34.25666 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/sinatra-2.2.4/lib/sinatra/base.rb:945:in `call' 2025-04-03_21:35:34.25666 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/gitlab-exporter-15.2.0/lib/gitlab_exporter/web_exporter.rb:51:in `call' 2025-04-03_21:35:34.25667 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/rack-2.2.11/lib/rack/logger.rb:17:in `call' 2025-04-03_21:35:34.25667 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/gitlab-exporter-15.2.0/lib/gitlab_exporter/web_exporter.rb:25:in `call' 2025-04-03_21:35:34.25667 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/rack-protection-2.2.4/lib/rack/protection/xss_header.rb:18:in `call' 2025-04-03_21:35:34.25667 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/rack-protection-2.2.4/lib/rack/protection/path_traversal.rb:16:in `call' 2025-04-03_21:35:34.25667 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/rack-protection-2.2.4/lib/rack/protection/json_csrf.rb:26:in `call' 2025-04-03_21:35:34.25667 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/rack-protection-2.2.4/lib/rack/protection/base.rb:50:in `call' 2025-04-03_21:35:34.25667 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/rack-protection-2.2.4/lib/rack/protection/base.rb:50:in `call' 2025-04-03_21:35:34.25668 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/rack-protection-2.2.4/lib/rack/protection/frame_options.rb:31:in `call' 2025-04-03_21:35:34.25668 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/rack-2.2.11/lib/rack/null_logger.rb:11:in `call' 2025-04-03_21:35:34.25668 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/rack-2.2.11/lib/rack/head.rb:12:in `call' 2025-04-03_21:35:34.25668 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/sinatra-2.2.4/lib/sinatra/show_exceptions.rb:22:in `call' 2025-04-03_21:35:34.25668 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/sinatra-2.2.4/lib/sinatra/base.rb:218:in `call' 2025-04-03_21:35:34.25668 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/sinatra-2.2.4/lib/sinatra/base.rb:2004:in `call' 2025-04-03_21:35:34.25668 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/sinatra-2.2.4/lib/sinatra/base.rb:1564:in `block in call' 2025-04-03_21:35:34.25668 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/sinatra-2.2.4/lib/sinatra/base.rb:1780:in `synchronize' 2025-04-03_21:35:34.25669 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/sinatra-2.2.4/lib/sinatra/base.rb:1564:in `call' 2025-04-03_21:35:34.25669 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/rack-2.2.11/lib/rack/handler/webrick.rb:95:in `service' 2025-04-03_21:35:34.25669 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/webrick-1.8.2/lib/webrick/httpserver.rb:140:in `service' 2025-04-03_21:35:34.25669 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/webrick-1.8.2/lib/webrick/httpserver.rb:96:in `run' 2025-04-03_21:35:34.25669 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/webrick-1.8.2/lib/webrick/server.rb:310:in `block in start_thread' 2025-04-03_21:35:34.27172 10.0.1.37 - - [03/Apr/2025:23:35:34 CEST] "GET /sidekiq HTTP/1.1" 500 184261 ``` In https://gitlab.com/gitlab-org/gitlab/-/issues/531486 and https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/8995 a similar error message during upgrades was reported, and a new `json` gem version was suspected to be the root cause. This new version `2.10.1` is also being called here in the exporter: ``` 2025-04-03_21:35:34.25659 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/json-2.10.1/lib/json/common.rb:245:in `parse' 2025-04-03_21:35:34.25659 /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/sidekiq-6.5.12/lib/sidekiq.rb:255:in `load_json' ``` ### What is the current *bug* behavior? The `gitlab-exporter` throws this error every 2-3 minutes, resulting in missing metrics in Prometheus: ``` $ cat /var/log/gitlab/gitlab-exporter | grep "InvalidByteSequence" 2025-04-04_05:50:49.25334 2025-04-04 07:50:49 - Encoding::InvalidByteSequenceError - "\xC3" on US-ASCII: 2025-04-04_05:54:34.25427 2025-04-04 07:54:34 - Encoding::InvalidByteSequenceError - "\xC3" on US-ASCII: 2025-04-04_05:55:19.25927 2025-04-04 07:55:19 - Encoding::InvalidByteSequenceError - "\xC3" on US-ASCII: 2025-04-04_06:05:34.26005 2025-04-04 08:05:34 - Encoding::InvalidByteSequenceError - "\xC3" on US-ASCII: 2025-04-04_06:05:49.26584 2025-04-04 08:05:49 - Encoding::InvalidByteSequenceError - "\xC3" on US-ASCII: 2025-04-04_06:08:49.26050 2025-04-04 08:08:49 - Encoding::InvalidByteSequenceError - "\xC3" on US-ASCII: 2025-04-04_06:10:34.25794 2025-04-04 08:10:34 - Encoding::InvalidByteSequenceError - "\xC3" on US-ASCII: 2025-04-04_06:10:49.25630 2025-04-04 08:10:49 - Encoding::InvalidByteSequenceError - "\xC3" on US-ASCII: 2025-04-04_06:15:19.26265 2025-04-04 08:15:19 - Encoding::InvalidByteSequenceError - "\xC3" on US-ASCII: 2025-04-04_06:15:34.26514 2025-04-04 08:15:34 - Encoding::InvalidByteSequenceError - "\xC3" on US-ASCII: 2025-04-04_06:20:34.26435 2025-04-04 08:20:34 - Encoding::InvalidByteSequenceError - "\xC3" on US-ASCII: 2025-04-04_06:21:19.25600 2025-04-04 08:21:19 - Encoding::InvalidByteSequenceError - "\xC3" on US-ASCII: 2025-04-04_06:24:19.25468 2025-04-04 08:24:19 - Encoding::InvalidByteSequenceError - "\xC3" on US-ASCII: 2025-04-04_06:25:19.25978 2025-04-04 08:25:19 - Encoding::InvalidByteSequenceError - "\xC3" on US-ASCII: 2025-04-04_06:25:34.25856 2025-04-04 08:25:34 - Encoding::InvalidByteSequenceError - "\xC3" on US-ASCII: 2025-04-04_06:25:49.25758 2025-04-04 08:25:49 - Encoding::InvalidByteSequenceError - "\xC3" on US-ASCII: 2025-04-04_06:26:19.25419 2025-04-04 08:26:19 - Encoding::InvalidByteSequenceError - "\xC3" on US-ASCII: 2025-04-04_06:29:19.26866 2025-04-04 08:29:19 - Encoding::InvalidByteSequenceError - "\xC3" on US-ASCII: 2025-04-04_06:30:19.26287 2025-04-04 08:30:19 - Encoding::InvalidByteSequenceError - "\xC3" on US-ASCII: 2025-04-04_06:33:04.25321 2025-04-04 08:33:04 - Encoding::InvalidByteSequenceError - "\xC3" on US-ASCII: 2025-04-04_06:35:19.25992 2025-04-04 08:35:19 - Encoding::InvalidByteSequenceError - "\xC3" on US-ASCII: 2025-04-04_06:35:34.28043 2025-04-04 08:35:34 - Encoding::InvalidByteSequenceError - "\xC3" on US-ASCII: 2025-04-04_06:38:34.25362 2025-04-04 08:38:34 - Encoding::InvalidByteSequenceError - "\xC3" on US-ASCII: ``` Because we are actively monitoring some of these metrics, we are getting many `NoData` alerts from Grafana because the data points are missing. ### What is the expected *correct* behavior? The `gitlab-exporter` should not throw this error and instead serve the metrics, as it was the case in GitLab v17.8.5. #### Results of GitLab environment info <details> <summary>Expand for output related to GitLab environment info</summary> <pre> System information System: Debian 11 Current User: git Using RVM: no Ruby Version: 3.2.5 Gem Version: 3.6.5 Bundler Version:2.6.5 Rake Version: 13.0.6 Redis Version: 7.0.15 Sidekiq Version:7.2.4 Go Version: unknown GitLab information Version: 17.10.3 Revision: 22d4014a923 Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: PostgreSQL DB Version: 14.17 URL: https://git.company.local HTTP Clone URL: https://git.company.local/some-group/some-project.git SSH Clone URL: git@git.company.local:some-group/some-project.git Using LDAP: yes Using Omniauth: yes Omniauth Providers: saml GitLab Shell Version: 14.41.0 Repository storages: - default: unix:/var/opt/gitlab/gitaly/gitaly.socket GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell Gitaly - default Address: unix:/var/opt/gitlab/gitaly/gitaly.socket - default Version: 17.10.3 - default Git Version: 2.48.1.gl1 </pre> </details> ### Possible fixes I assume the upgraded `json` gem is the root cause for this, maybe a downgrade would be useful?

issue