Sidekiq import/export logs use Rails.logger which does not use structured logging
When project exports fail we log a failure message using Rails.logger, on omnibus installations this ends up in production.log
which was recently removed from central logging. It would be much better for us to have these logs in the production_json.json
structured log or a separate sidekiq structured logfile.
This limits visibility into failures because the from the sidekiq log we see this failed job as completed, but without any more information. see example: gitlab-com/gl-infra/scalability#200 (comment 298941726)
Designs
- Show closed items
Activity
-
Newest first Oldest first
-
Show all activity Show comments only Show history only
- John Jarvis added gitlab.com production request labels
added gitlab.com production request labels
- John Jarvis mentioned in issue gitlab-com/gl-infra/scalability#200
mentioned in issue gitlab-com/gl-infra/scalability#200
- John Jarvis changed title from Sidekiq import export logs use Rails.logger which to Sidekiq import/export logs use Rails.logger which does not use structured logging
changed title from Sidekiq import export logs use Rails.logger which to Sidekiq import/export logs use Rails.logger which does not use structured logging
- Owner
I'm adding infradev label because we do not have any visibility into what the project export is doing. We do not know whether we are experiencing any issues with the feature.
- Maintainer
Setting devopsmanage based on ~"group::import".
- 🤖 GitLab Bot 🤖 added devopsmanage label
added devopsmanage label
- Christopher Lefelhocz assigned to @lmcandrew
assigned to @lmcandrew
- Contributor
@lmcandrew This feels like an easy one (famous last words). Can we get this scheduled as part of 12.10?
Collapse replies @lmcandrew Follow up if this one is ready to be scheduled.
@lmcandrew Please circle back current status of this issue when you get chance.
- Contributor
~"group::import" will consider this a candidate for %12.10.
- Josianne Hyson mentioned in issue #121732 (closed)
mentioned in issue #121732 (closed)
- Josianne Hyson mentioned in merge request !27114 (closed)
mentioned in merge request !27114 (closed)
- Haris Delalić added Category:Importers backend workflowscheduling labels
added Category:Importers backend workflowscheduling labels
- Haris Delalić changed milestone to %12.10
changed milestone to %12.10
- Haris Delalić added estimationneeded label
added estimationneeded label
- Haris Delalić added to epic &2959
added to epic &2959
- Kassio Borges added workflowready for development label and removed workflowscheduling label
added workflowready for development label and removed workflowscheduling label
- Liam McAndrew unassigned @lmcandrew
unassigned @lmcandrew
- 🤖 GitLab Bot 🤖 added [deprecated] Accepting merge requests label
added [deprecated] Accepting merge requests label
- Josianne Hyson changed health status to at risk
changed health status to at risk
- Haris Delalić changed milestone to %13.0
changed milestone to %13.0
- Haris Delalić added milestonep2 label
added milestonep2 label
- Haris Delalić removed the health status
removed the health status
- Haris Delalić added workflowblocked label and removed workflowready for development label
added workflowblocked label and removed workflowready for development label
- Haris Delalić added workflowready for development label and removed workflowblocked label
added workflowready for development label and removed workflowblocked label
- Haris Delalić added Deliverable label
added Deliverable label
- Contributor
@jhyson @kassio @georgekoltsov @xanf: Can you please provide your weight estimate?
2 1 Collapse replies - Developer
I would say a
as it should be fairly straightforward to find all the locations that are usingRails.logger
and swap them out. - Maintainer
I agree, I think
should be fine.
- Haris Delalić set weight to 2
set weight to 2
- Haris Delalić added estimationcompleted label and removed estimationneeded label
added estimationcompleted label and removed estimationneeded label
- Haris Delalić changed milestone to %13.1
changed milestone to %13.1
- George Koltsov added workflowin dev label and removed workflowready for development label
added workflowin dev label and removed workflowready for development label
- George Koltsov assigned to @georgekoltsov
assigned to @georgekoltsov
- 🤖 GitLab Bot 🤖 removed [deprecated] Accepting merge requests label
removed [deprecated] Accepting merge requests label
- Maintainer
@jarv quick question on importer log file. Is it safe to rename existing log file defined here? https://gitlab.com/gitlab-org/gitlab/-/blob/8cc37ed4bc373c417d890166a2a0b5d6ee2fc8ae/lib/gitlab/import/logger.rb#L7
Right now we have
importer.log
which also has export logs coming from Group Export. Would there be any unexpected side effects with renamingimporter.log
into sayimport_export.log
? Or would it be better to create a newexporter.log
? We will have to update docs in both cases. Just not sure if there is something specific that will have to be done for omnibus installations.Edited by George Koltsov Collapse replies - Author Developer
For queues that run in kubernetes it doesn't matter because we glob
/var/log/gitlab/*
and send everything to a single index. Currently onlyproject_export
is running in kubernetes however.If this affects projects import as well, we should add this new logfile to our fluentd configuration, by adding a new section for https://gitlab.com/gitlab-cookbooks/gitlab_fluentd/-/blob/master/templates/default/rails.conf.erb#L62-75
There is no harm in leaving
importer.log
in the fluentd config as-is, so we just need to copy-paste a new block. 1 - Maintainer
Thanks @jarv I will create json
exporter.log
similar to importer one and move all export related log lines there.I was trying to create an MR in gitlab_fluentd, but don't have permissions. Can you add me to this project or should I go via access-request route?
- Author Developer
@georgekoltsov you should have access now.
1
- George Koltsov assigned to @jarv
assigned to @jarv
- George Koltsov added workflowin review label and removed workflowin dev label
added workflowin review label and removed workflowin dev label
- George Koltsov unassigned @jarv
unassigned @jarv
- George Koltsov mentioned in merge request gitlab-cookbooks/gitlab_fluentd!156 (merged)
mentioned in merge request gitlab-cookbooks/gitlab_fluentd!156 (merged)
- George Koltsov mentioned in merge request !32543 (merged)
mentioned in merge request !32543 (merged)
- George Koltsov mentioned in merge request gitlab-com/runbooks!2275 (merged)
mentioned in merge request gitlab-com/runbooks!2275 (merged)
- George Koltsov added workflowverification label and removed workflowin review label
added workflowverification label and removed workflowin review label
- Maintainer
Closing this issue as changes reached GitLab.com -- all Rails.logger occurrences have been replaced with structured logging for Import/Export.
- George Koltsov closed
closed
- George Koltsov added workflowproduction label and removed workflowverification label
added workflowproduction label and removed workflowverification label
- Haris Delalić added Importer:Project Export/Import label
added Importer:Project Export/Import label
- 🤖 GitLab Bot 🤖 added groupimport and integrate label and removed 1 deleted label
added groupimport and integrate label and removed 1 deleted label