errors from Ci::AppendBuildTraceService don't seem to be shipping with the correct exception.message
Summary
If a self-managed instance needs to enable incremental logging because there are
- multiple Rails nodes
- object storage
- no shared filesystem
Job logs will get lost.
They'll be written to the filesystem on one of the Rails nodes, but Sidekiq won't be able to locate them on other nodes when it comes to archiving them to object storage.
One symptom of this is the following error in the Rails exceptions_json.log
, which I think is emitted by the Ci::AppendBuildTraceService
class
{
"severity": "ERROR",
"time": "YYYY-MM-DDT15:12:48.610Z",
"correlation_id": "xx",
"exception.class": "Ci::AppendBuildTraceService::TraceRangeError",
"exception.message": "Ci::AppendBuildTraceService::TraceRangeError",
"user.username": "xxx",
"tags.program": "web",
"tags.locale": "en",
"tags.feature_category": "continuous_integration",
"tags.correlation_id": "xx"
"extra.build_id": 425187,
"extra.body_end": 12955,
"extra.stream_size": 720,
"extra.stream_class": {},
"extra.stream_range": "0-12954"
}
Question: is this how it's expected to be formatted? The exception class is being written to exception.message
rather than anything describing the error that occurred.
Note: this was observed on a 14.6. I don't see any obvious change to how exceptions are logged, but if material changes have been made since then, this might invalidate this issue.
Steps to reproduce
- Run an instance with at least two Rails nodes; configure object storage, no shared (eg: NFS storage).
- Run CI jobs.
- Wait for the nodes to attempt to read build traces that were written to local storage on the other node.
Example Project
What is the current bug behavior?
The exception class is being written to exception.message
What is the expected correct behavior?
exception.message
contains something describing the error that occurred.
Relevant logs and/or screenshots
Output of checks
Results of GitLab environment info
Expand for output related to GitLab environment info
(For installations with omnibus-gitlab package run and paste the output of: `sudo gitlab-rake gitlab:env:info`) (For installations from source run and paste the output of: `sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production`)
Results of GitLab application Check
Expand for output related to the GitLab application check
(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:check SANITIZE=true
)(For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true
)(we will only investigate if the tests are passing)