Investigate "Failed to log to appender" errors
We've been seeing errors like the following in release-tools CI job logs for many months now. However, the logs do get uploaded to ElasticSearch since they are visible in https://nonprod-log.gitlab.net/.
This issue is for investigating the error.
Job logs from https://ops.gitlab.net/gitlab-org/release/tools/-/jobs/12864336:
2024-02-16 00:46:47.684337 E [17:SemanticLogger::Appenders] SemanticLogger::Appenders -- Failed to log to appender: SemanticLogger::Appender::ElasticsearchHttp -- Exception: Net::ReadTimeout: Net::ReadTimeout with #<TCPSocket:(closed)>
/usr/local/lib/ruby/3.2.0/net/protocol.rb:229:in `rbuf_fill'
/usr/local/lib/ruby/3.2.0/net/protocol.rb:199:in `readuntil'
/usr/local/lib/ruby/3.2.0/net/protocol.rb:209:in `readline'
/usr/local/lib/ruby/3.2.0/net/http/response.rb:158:in `read_status_line'
/usr/local/lib/ruby/3.2.0/net/http/response.rb:147:in `read_new'
/usr/local/lib/ruby/3.2.0/net/http.rb:2342:in `block in transport_request'
/usr/local/lib/ruby/3.2.0/net/http.rb:2333:in `catch'
/usr/local/lib/ruby/3.2.0/net/http.rb:2333:in `transport_request'
/usr/local/lib/ruby/3.2.0/net/http.rb:2306:in `request'
/usr/local/bundle/gems/sentry-ruby-5.16.1/lib/sentry/net/http.rb:30:in `request'
/usr/local/bundle/gems/semantic_logger-4.15.0/lib/semantic_logger/appender/http.rb:234:in `process_request'
/usr/local/bundle/gems/semantic_logger-4.15.0/lib/semantic_logger/appender/http.rb:213:in `post'
/usr/local/bundle/gems/semantic_logger-4.15.0/lib/semantic_logger/appender/elasticsearch_http.rb:70:in `log'
/usr/local/bundle/gems/semantic_logger-4.15.0/lib/semantic_logger/appenders.rb:31:in `block in log'
/usr/local/bundle/gems/semantic_logger-4.15.0/lib/semantic_logger/appenders.rb:30:in `each'
/usr/local/bundle/gems/semantic_logger-4.15.0/lib/semantic_logger/appenders.rb:30:in `log'
/usr/local/bundle/gems/semantic_logger-4.15.0/lib/semantic_logger/appender/async.rb:152:in `process_messages'
/usr/local/bundle/gems/semantic_logger-4.15.0/lib/semantic_logger/appender/async.rb:121:in `process'
/usr/local/bundle/gems/semantic_logger-4.15.0/lib/semantic_logger/appender/async.rb:77:in `block in thread'