N+1s in PipelineDetailsEntity when serializing multiple TriggeredPipelineEntity
Problem
from !35232 (comment 368325716)
We don't preload properly the
triggered_pipelines
andtriggered_by_pipelines
. The97
number is what comes up today prior to introducing thesource_job
. In the existing spec examples where we check the number of queries we use a simple pipeline. When serializing a pipeline with upstream or (worse) downstream pipelines we introduce N+1s.From inspecting the recorded log it seems that the majority of the queries are related to routing when building the project urls but there is also other data not being preloaded properly. Things can be worse if we use the
expanded
version of the serializer, where for each downstream pipeline we fetch their own upstream/downstream pipelines.
Solution
- preload properly data used to serialize
triggered_by_pipeline
andtriggered_pipelines
. - ensure that with
expanded
serializer we can still reduce the number of queries being fired.
Edited by Fabio Pitino