Project 'gitlab-com/gl-infra/scalability' was moved to 'gitlab-com/gl-infra/observability/team'. Please update any links and bookmarks that may still have the old path.
Ingest sidekiq_client.log in the pubsub-sidekiq-inf-gprd index
Perhaps after we included this file, we could have change the Sidekiq::StructuredLogger to write to the Gitlab::SidekiqLogger to have all of the sidekiq related stuff be in one place on all nodes.
so we think when this is implemented we will have something like /var/log/gitlab/gitlab-rails/sidekiq_json.log on omnibus installations?
@jarv Yes that is the intention. In after #200 (closed) is implemented /var/log/gitlab/gitlab-rails/sidekiq_json.log & /var/log/gitlab/gitlab-rails/sidekiq.log would be written to. But we should only care about the structured one.
Bob Van Landuytchanged title from Absorb sidekiq_json.log in the pubsub-sidekiq-inf-gprd index to Absorb sidekiq_client.log in the pubsub-sidekiq-inf-gprd index
changed title from Absorb sidekiq_json.log in the pubsub-sidekiq-inf-gprd index to Absorb sidekiq_client.log in the pubsub-sidekiq-inf-gprd index
I'll move this into ~"workflow::Ready" now that the sidekiq_client.log has been created and is being written to.
Bob Van Landuytchanged title from Absorb sidekiq_client.log in the pubsub-sidekiq-inf-gprd index to Ingest sidekiq_client.log in the pubsub-sidekiq-inf-gprd index
changed title from Absorb sidekiq_client.log in the pubsub-sidekiq-inf-gprd index to Ingest sidekiq_client.log in the pubsub-sidekiq-inf-gprd index
On the left, we're seeing the start messages, from before we flipped the switch, we can clearly see a lot of messages with caller id from the API.
The graph on the right is supposed to pick up where the graph on the left stops: It's showing the deduplicated messages, but we can clearly see the ones with caller_ids from the API are missing. Because these messages are added to the sidekiq_client.log on the API fleet.
The deduplicated messages from the sidekiq fleet go to the same logfile as the others, so it's no surprise they're there.
I'll ask someone with access to SSH into one of those node to tail /var/log/gitlab/gitlab-rails/sidekiq_client.log and see what's happening.
Yeah, so I didn't fully understand how some of our fluentd stuff worked and assumed the 'tag' was all that was necessary for routing messages. I ran a little experiment on api-01 and got it working (so there's a few logs in there from 02:00-02:10UTC on 2020-04-03). I need to refactor our fluentd configuration slightly so that it works as I expected; sorry about this. Not sure if I'll get it out today as the change is non-trivial (not awful, just not a simple addition) so I'll need eyes-on before I roll it out. I'll add a link when I have the MR prepared.
Rolled it out to a couple of nodes manaully and letting chef pick it up on the rest over the next ~30 minutes. We have logs now, and the visualization is seeing /api/:version/jobs/:id/artifacts entries now.