Skip to content

Transfer data blueprint

Vasilii Iakliushin requested to merge add_transfer_data_blueprint_take_2 into master

What does this MR do and why?

Contributes to https://gitlab.com/groups/gitlab-org/-/epics/7421

Create a blueprint for Transfer Data feature.

It covers the overall design and implementation only for Repository Egress.

Diagram from the document

flowchart TB

   A[Applications] -->|send logs| Pub(Google Pub/Sub)
   Pub -->|subscribe to topic|JSONParser
   

   subgraph DataFlowPipeline
      direction TB

      JSONParser -->|selects only JSON lines| LogProcessor
      LogProcessor -->|insert detected data transfer logs|ClickHouse
   end

ClickHouse -->|query transfer logs| Rails

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Vasilii Iakliushin

Merge request reports