Projects::JobsController#raw times out with many over 1,500 HTTP calls
In https://log.gprd.gitlab.net/goto/aee2f2f0-c229-11ed-8afc-c9851e4645c0, I see a number of timeouts retrieving job traces, sometimes requiring over 1,500 HTTP calls:
"exception.backtrace": [
"app/models/ci/build_trace_chunks/fog.rb:19:in `data'",
"app/models/ci/build_trace_chunk.rb:217:in `get_data'",
"app/models/ci/build_trace_chunk.rb:100:in `data'",
"lib/gitlab/ci/trace/chunked_io.rb:196:in `block in chunk_slice_from_offset'",
"lib/gitlab/ci/trace/chunked_io.rb:193:in `chunk_slice_from_offset'",
"lib/gitlab/ci/trace/chunked_io.rb:77:in `read'",
"lib/gitlab/ci/trace/stream.rb:62:in `raw'",
"app/controllers/projects/jobs_controller.rb:151:in `block in raw'",
"lib/gitlab/ci/trace.rb:166:in `read_stream'",
"lib/gitlab/ci/trace.rb:88:in `read'",
"app/controllers/projects/jobs_controller.rb:142:in `raw'",
One of the job logs was 54 MB with 718,001 lines--didn't seen that out of line.
I suspect there were too many HTTP calls here.