Transfer data blueprint
What does this MR do and why?
Contributes to https://gitlab.com/groups/gitlab-org/-/epics/7421
Create a blueprint for Transfer Data feature.
It covers the overall design and implementation only for Repository Egress.
Diagram from the document
flowchart TB
A[Applications] -->|send logs| Pub(Google Pub/Sub)
Pub -->|subscribe to topic|JSONParser
subgraph DataFlowPipeline
direction TB
JSONParser -->|selects only JSON lines| LogProcessor
LogProcessor -->|insert detected data transfer logs|ClickHouse
end
ClickHouse -->|query transfer logs| Rails
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.
Edited by Vasilii Iakliushin