Add ability to store CI trace chunks on Gitlab::Redis::TraceChunks [RUN ALL RSPEC] [RUN AS-IF-FOSS]
What does this MR do?
Implements gitlab-com/gl-infra/scalability#1095 (closed)
GitLab has an optional feature to store unfinished CI job logs in Redis rather than on NFS: Incremental job logs. In gitlab-com/gl-infra&462 (closed) we are working on migrating GitLab.com to use a dedicated Redis instance just for the unfinished job logs, called Gitlab::Redis::TraceChunks.
In the application, unfinished job logs are implemented using the Ci::BuildTraceChunk ActiveRecord type which is stored in the ci_build_trace_chunks Postgres table. The ci_build_trace_chunks already has a column data_store which indicates where the trace chunk data is stored. To implement the change we introduce a new enum value for the data_store column which indicates the trace chunk data is in Gitlab::Redis::TraceChunks rather than Gitlab::Redis::SharedState (the old location).
Note that if the administrator does not explicitly configure Gitlab::Redis::TraceChunks via its dedicated config file redis.trace_chunks.yml, it will point to the same Redis server as Gitlab::Redis::SharedState.
This MR adds a feature flag dedicated_redis_trace_chunks which controls the Redis instance for new trace chunks. The default value of the flag is false which causes new trace chunks to be stored on Gitlab::Redis::SharedState, the old location. When we change the feature flag to true new trace chunks will start using
Gitlab::Redis::TraceChunks. Although we do not strictly need the feature flag from a functional perspective (we could do a correct roll-out using MR's alone) it is desirable to use it from an operational perspective because we know that trace chunks create a lot of Redis traffic on gitlab.com and it would be better to move that over gradually, while someone is watching, than as a side effect of a production deploy.
Screenshots (strongly suggested)
Does this MR meet the acceptance criteria?
Conformity
-
I have included a changelog entry, or it's not needed. (Does this MR need a changelog?) -
I have added/updated documentation, or it's not needed. (Is documentation required?) -
I have properly separated EE content from FOSS, or this MR is FOSS only. (Where should EE code go?) -
I have added information for database reviewers in the MR description, or it's not needed. (Does this MR have database related changes?) -
I have self-reviewed this MR per code review guidelines. -
This MR does not harm performance, or I have asked a reviewer to help assess the performance impact. (Merge request performance guidelines) -
I have followed the style guides.
Availability and Testing
-
I have added/updated tests following the Testing Guide, or it's not needed. (Consider all test levels. See the Test Planning Process.) -
I have tested this MR in all supported browsers, or it's not needed. -
I have informed the Infrastructure department of a default or new setting change per definition of done, or it's not needed.
Security
Does this MR contain changes to processing or storing of credentials or tokens, authorization and authentication methods or other items described in the security review guidelines? If not, then delete this Security section.
-
Label as security and @ mention @gitlab-com/gl-security/appsec -
The MR includes necessary changes to maintain consistency between UI, API, email, or other methods -
Security reports checked/validated by a reviewer from the AppSec team