WIP: New CI Job live-trace architecture (CE)

• CE: Here
• EE: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/5110

What does this MR do?

• This is implemented on top of this proposal.
• This is implemented as minimal as possible.
• Gitlab::Ci::Trace::ChunkedFile::ChunkedIO is an abstract layer for manipulating chunked files. It's compatible with ruby's IO class.
• Gitlab::Ci::Trace::ChunkedFile::ChunkedIO::LiveTrace is for manipulating live-trace concerned operations. It uses ChunkStore::Redis and ChunkStore::Database for storing pieces of traces which sent from gitlab-runners.
• Gitlab::Ci::Trace::ChunkedFile::ChunkStore is for storing chunks
• Gitlab::Ci::Trace::ChunkedFile::ChunkStore::Redis is for writing/appending piece of chunks
• Gitlab::Ci::Trace::ChunkedFile::ChunkStore::Database is for stashing complete chunks from ChunkStore::Redis
• The buffer size of each store is 128 kb.
• This is to be deployed to staging server and check the impact to the system.
• This is easy to extend for another solution, for example, Redis+ObjectStorage combo.

TODOs

• Each write/archive operation should be wrapped by Sidekiq for scaling horizontally

Are there points in the code the reviewer needs to double check?

• Is there any performance regressions?
• What if Redis is not responded?
• What if Postgres is not responded?
• Can we archive the complete trace without data loss?

Why was this MR needed?

This is necessary for ~"Cloud Native" compatible gitlab.

Screenshots (if relevant)

Does this MR meet the acceptance criteria?

• Changelog entry added, if necessary
• Tests added for this feature/bug
• Review
  • Has been reviewed by Backend
  • Has been reviewed by Database
• End-to-end tests pass (package-qa manual pipeline job)

What are the relevant issue numbers?

Close https://gitlab.com/gitlab-org/gitlab-ee/issues/4607