Skip to content

Allow operations to be instrumented once in Pipeline creation logger

Fabio Pitino requested to merge pipeline-creation-observe-once into master

What does this MR do and why?

Related to Pipeline step log entries have too many fields (#371636 - closed)

The pipeline creation logger currently logs too many fields to Elasticsearch causing performance and stability issues to the stack. We need to reduce the number of fields and remove those that are not essential.

Some of the observations used in the logger are only occurring once per log instance. For example: pipeline_creation_duration_s is observed only once at the end of the pipeline creation process. For such observations we don't need to have multiple values.

This would reduce further the number of fields indexed in Elasticsearch because pipeline_creation_duration_s.sum and pipeline_creation_duration_s.count simply become 1 field: pipeline_creation_duration_s. In fact for these fields the .count will always be 1.

Screenshots or screen recordings

Screenshots are required for UI changes, and strongly recommended for all other merge requests.

How to set up and validate locally

Numbered steps to set up and validate the change are strongly suggested.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Marius Bobin

Merge request reports