Skip to content

Extra logging for new live trace architecture

Shinya Maeda requested to merge safe-archiving-for-traces into master

What does this MR do?

https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/4667#note_192604972

  • This happens when live trace is archived.
  • The live trace which stored in Redis and Object Storage are correctly pulled into gitlab-rails. It successfully generates pre-archived trace file in temporary dir.
  • CarreirWave fails to upload the pre-archived trace file to the object storage, thus gitlab-rails fails to create ci_job_artifacts record as archived trace, however, the system thinks it's successfully archived and removing live traces.
  • As the result, there are no live nor archive traces.

Also, for debug purpose, we print job id into the temporary directory path for pre-archived files. This way, we can verify that missing traces are corresponding to the leftover traces in the temporary directory. Since it's temporary directory, there are no harm to change the path.

There are three possibilities when trace is missing.

  1. Live Trace didn't exist at all.
  2. Live Trace was deleted before archive.
  3. Archived trace was delete without intention.

We add extra logging to distinguish in which step the culprit resides.

Does this MR meet the acceptance criteria?

Conformity

Performance and testing

Security

If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:

  • Label as security and @ mention @gitlab-com/gl-security/appsec
  • The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
  • Security reports checked/validated by a reviewer from the AppSec team
Edited by Shinya Maeda

Merge request reports