2024-02-21: elevated error rates for gprd-cny

added an incident timeline event

added IncidentActive Source::IMAIncidentDeclare blocks deployments incident severity3 labels

assigned to @ayeung

changed the description

changed the severity to Medium - S3

added a resource link

@schin1 reported elevated error rates for gprd-cny in Slack: https://gitlab.slack.com/archives/C8PKBH3M5/p1708474749851039?thread_ts=1708470230.490199&cid=C8PKBH3M5

Info so far:

deployment of 16.10.202402202010-9c91ec7b214.407c35432a8 in gprd-cny started the climb in error rate on gprd-cny
web service error rates increasing steadily from that deployment on cny (dashboard)
sentry shows Encoding::CompatibilityError (https://new-sentry.gitlab.net/organizations/gitlab/releases/9c91ec7b214/?project=3) for that release, affecting a bunch of controllers
logs shows increase in 5xx too, especially for Projects::RawController#show

I haven't been able to reproduce this with an Omnibus install and a Cloud Native GitLab with that version.

I even issued a curl -H "Cookie: _gitlab_session=<my cookie>" http://localhost:8080/-/asdf on a cny pod running a779caa4045, and the 404 returned fine.

I'm wondering if only spiders are able to trigger this with some input.

I noticed that Rails was loading the errors.css as ASCII-8BIT:

[ gprd ] production> Rails.application.assets_manifest.find_sources('errors.css').first.to_s.html_safe.encoding
=> #<Encoding:ASCII-8BIT>

I copied good.css from a v16.8.0 errors*.css, and the bad.css from registry.gitlab.com/gitlab-org/security/charts/components/images/gitlab-webservice-ee:16-10-202402202010-9c91ec7b214:

docker pull registry.gitlab.com/gitlab-org/security/charts/components/images/gitlab-webservice-ee:16-10-202402202010-9c91ec7b214
docker run --name bad -it registry.gitlab.com/gitlab-org/security/charts/components/images/gitlab-webservice-ee:16-10-202402202010-9c91ec7b214 bash

In another window:

docker cp bad:/srv/gitlab/public/assets/errors-03910abb66ddd056528b660304c864789459749f77453589d51036fe83101f91.css /tmp/bad.css

I noticed that bad.css contains a UTF-8 character, unlike the previous good.css:

$ file bad.css
bad.css: Unicode text, UTF-8 text, with very long lines (55179)
$ file good.css
good.css: ASCII text, with very long lines (55432)

Where is this UTF-8 character?

$ fold -w 80 bad.css > bad-formatted.css
$ grep -P "[^\x00-\x7F]" bad-formatted.css
before{content:"·";display:inline-block;padding:0 1em}}.tanuki-logo{width:210px

Notice the presence of the · character.

I could see an issue if Rails is loading the errors.css incorrectly as ASCII-8BIT, and then tries to add UTF-8 data with it.

bad.css good.css

Ok, I reproduced the error by changing my language preference to Chinese, then triggering a 404 by visiting a non-existent URL.

On the Omnibus instance with this version:

irb(main):001:0> Rails.application.assets_manifest.class
=> Sprockets::Manifest
irb(main):002:0> error = Rails.application.assets_manifest.find_sources('errors.css').first.to_s.html_safe
=> "*,*::before,*::after{box-sizing:border-box}html{font-family:sans-serif;line-height:1.15;-webkit-text-size-adjust:100%;-webkit-tap-hig...
irb(main):003:0> "#{error} 中國"
(irb):3:in `<main>': incompatible character encodings: ASCII-8BIT and UTF-8 (Encoding::CompatibilityError)

Sprockets::Manifest loads the source via File.binread in https://github.com/rails/sprockets/blob/d1dcf7075c468522e1cb6f93ae547d8d7fdcfbcb/lib/sprockets/asset.rb#L99, so the CSS will always be loaded as ASCII-8BIT. When ASCII-8BIT combines with UTF-8 characters,

I have also managed to reproduce it locally using GDK:

Add config.assets.compile = false to config/environments/development.rb
Precompile the assets by running bin/rake gitlab:assets:compile
Visit a page that 404s (e.g. http://localhost:3000/gitlab-org/gitlab/-/issues/undefined)

The steps are required because Sprockets does not call File.binread if compilation is enabled.

Draining canary now.

cny is drained: https://gitlab.slack.com/archives/C101F3796/p1708475535443309

Full stack trace from correlation_id : 01HQ4H0FD1CHJ2GVF76V4KCVBZ:

json.exception.cause_class: ExtractsRef::RefExtractor::InvalidPathError
json.exception.class: Encoding::CompatibilityError

app/views/layouts/errors.html.haml:11
 app/controllers/application_controller.rb:132:in `render'
 app/controllers/application_controller.rb:248:in `block (2 levels) in render_404'
 app/controllers/application_controller.rb:247:in `render_404'
 lib/extracts_path.rb:48:in `rescue in assign_ref_vars'
 lib/extracts_path.rb:36:in `assign_ref_vars'
 ee/lib/gitlab/ip_address_state.rb:10:in `with'
 ee/app/controllers/ee/application_controller.rb:45:in `set_current_ip_address'
 app/controllers/application_controller.rb:468:in `set_current_admin'
 lib/gitlab/session.rb:11:in `with_session'
 app/controllers/application_controller.rb:459:in `set_session_storage'
 lib/gitlab/i18n.rb:114:in `with_locale'
 app/controllers/application_controller.rb:452:in `set_locale'
 app/controllers/application_controller.rb:443:in `set_current_context'
 ee/lib/omni_auth/strategies/group_saml.rb:41:in `other_phase'
 lib/gitlab/metrics/elasticsearch_rack_middleware.rb:16:in `call'
 lib/gitlab/middleware/memory_report.rb:13:in `call'
 lib/gitlab/middleware/speedscope.rb:13:in `call'
 lib/gitlab/database/load_balancing/rack_middleware.rb:23:in `call'
 lib/gitlab/middleware/rails_queue_duration.rb:33:in `call'
 lib/gitlab/etag_caching/middleware.rb:21:in `call'
 lib/gitlab/metrics/rack_middleware.rb:16:in `block in call'
 lib/gitlab/metrics/web_transaction.rb:46:in `run'
 lib/gitlab/metrics/rack_middleware.rb:16:in `call'
 lib/gitlab/middleware/go.rb:20:in `call'
 lib/gitlab/middleware/query_analyzer.rb:11:in `block in call'
 lib/gitlab/database/query_analyzer.rb:40:in `within'
 lib/gitlab/middleware/query_analyzer.rb:11:in `call'
 lib/gitlab/middleware/organizations/current.rb:24:in `call'
 lib/gitlab/middleware/multipart.rb:173:in `call'
 lib/gitlab/middleware/read_only/controller.rb:50:in `call'
 lib/gitlab/middleware/read_only.rb:18:in `call'
 lib/gitlab/middleware/unauthenticated_session_expiry.rb:18:in `call'
 lib/gitlab/middleware/same_site_cookies.rb:27:in `call'
 lib/gitlab/middleware/path_traversal_check.rb:35:in `call'
 lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'
 lib/gitlab/middleware/basic_health_check.rb:25:in `call'
 lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'
 lib/gitlab/middleware/request_context.rb:15:in `call'
 lib/gitlab/middleware/webhook_recursion_detection.rb:15:in `call'
 config/initializers/fix_local_cache_middleware.rb:11:in `call'
 lib/gitlab/middleware/compressed_json.rb:44:in `call'
 lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call'
 lib/gitlab/middleware/sidekiq_web_static.rb:20:in `call'
 lib/gitlab/metrics/requests_rack_middleware.rb:79:in `call'
 lib/gitlab/middleware/release_env.rb:13:in `call'