Extra logging for new live trace architecture
What does this MR do?
https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/4667#note_192604972
- This happens when live trace is archived.
- The live trace which stored in Redis and Object Storage are correctly pulled into gitlab-rails. It successfully generates pre-archived trace file in temporary dir.
- CarreirWave fails to upload the pre-archived trace file to the object storage, thus gitlab-rails fails to create
ci_job_artifacts
record as archived trace, however, the system thinks it's successfully archived and removing live traces.- As the result, there are no live nor archive traces.
Also, for debug purpose, we print job id into the temporary directory path for pre-archived files. This way, we can verify that missing traces are corresponding to the leftover traces in the temporary directory. Since it's temporary directory, there are no harm to change the path.
There are three possibilities when trace is missing.
- Live Trace didn't exist at all.
- Live Trace was deleted before archive.
- Archived trace was delete without intention.
We add extra logging to distinguish in which step the culprit resides.
Does this MR meet the acceptance criteria?
Conformity
-
Changelog entry for user-facing changes, or community contribution. Check the link for other scenarios. -
Documentation created/updated or follow-up review issue created -
Code review guidelines -
Merge request performance guidelines -
Style guides -
Database guides -
Separation of EE specific content
Performance and testing
-
Review and add/update tests for this feature/bug. Consider all test levels. See the Test Planning Process. -
Tested in all supported browsers
Security
If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:
-
Label as security and @ mention @gitlab-com/gl-security/appsec
-
The MR includes necessary changes to maintain consistency between UI, API, email, or other methods -
Security reports checked/validated by a reviewer from the AppSec team
Edited by Shinya Maeda