Have a dedicated `pipeline` logger for tracking pipeline activity/life cycle.
Problem
Given the life cycle of pipeline is quite complicated today - which depend on multiple asynchronous processes, sidekiq and runners, we have some race conditions that we have no idea what's going on like recent data loss on job traces. We should have a dedicated logger for monitoring pipeline/job activity to make sure the transition is correctly performed in any circumstances.
This is basically done by Application logger or Sidekiq logger today, however, given it's split per instance nodes, it's hard to trace a specific context. That said, we should introduce per-context-log then per-instance-log.
Proposal
Introduce a new log file pipeline.json (PipelineLogger
). Basically, we should log the following things:
- Pipeline related state transition (e.g. Job/Trace/Artifact)
- Warning on exceptional cases (e.g. !17043 (diffs))
JSON structure
{
"severity": <The severity>,
"time": <Timestamp>,
"correlation_id": <Correlation ID>,
"pipeline_id": <The ID of the pipeline>,
"job_id": <The ID of the pipeline job>,
"message": <The message>
}
Edited by Shinya Maeda