Skip to content

Swap redis chunks to database when jobs finished (for `dependent: :destory` problem)

Shinya Maedarequested to merge
live-trace-v2-schedule-db-when-job-done into live-trace-v2

What does this MR do?

  • This MR is for resolving dependent: destory problem
  • This MR adds a process to swap Redis chunks to Database when job finished
  • The fastest worker(BuildFinishedWorker with pipeline_processing queue) will swap data from Redis to Database before archving. This worker is scheduled when job finished
  • We have the situation that Sidekiq worker is unstable, however, we also don't take into account that it takes BuildFinishedWorker will never be
  • Ci::JobTraceChunk::CHUNK_REDIS_TTL = 1.week. We compromise that the garbage chunks remain by some cases (e.g. when a project is removed while a job is running). But it will be cleaned after 1 week. Maybe the TTL can be shorter because this MR clears one of the problems that ArchiveTraceWorker takes long time sometimes.
  • This MR is to be merged to live-trace-v2 instead of master
  • This idea was described in https://gitlab.com/gitlab-org/gitlab-ce/issues/44935#note_69282413

There are alternative approaches

Are there points in the code the reviewer needs to double check?

Why was this MR needed?

Screenshots (if relevant)

Does this MR meet the acceptance criteria?

  • Tests added for this feature/bug
  • Review
    • Has been reviewed by Backend
    • Has been reviewed by Database
  • End-to-end tests pass (package-and-qa manual pipeline job)

What are the relevant issue numbers?

Edited by Shinya Maeda

Merge request reports