We need to ship a PoC demonstrating that we can support cascading partition_id through Rails associations, resulting in partitioning key being used when a method added by Rails associations is called.
[1]pry(main)>ActiveRecord::Base.logger=Logger.new(STDOUT);Ci::Pipeline.first.builds.countD,[2022-04-20T11:49:23.506227#184] DEBUG -- : Ci::Pipeline Load (1.1ms) SELECT "ci_pipelines".* FROM "ci_pipelines" ORDER BY "ci_pipelines"."id" ASC LIMIT 1D,[2022-04-20T11:49:23.543371#184] DEBUG -- : (1.2ms) SELECT COUNT(*) FROM "ci_builds" WHERE "ci_builds"."type" = 'Ci::Build' AND "ci_builds"."commit_id" = 1 AND "ci_builds"."partition_id" = 100=>16
It is using a partition_id assigned to a pipeline resource, that we decided to cascade into child resources, like builds / stages.
We might be able to add has_many_partitioned :builds scope to hide this implementation detail later. I wonder if we should extend this PoC with this
All that being said, I believe that this PoC proves that this can be done, and this potential detail should not make partition CI/CD resources more difficult / challenging.
@grzesiek what about preloads? I think Ci::Pipeline.where(something).includes(:builds) will fail. I don't know if we actually use it anywhere because searching for \.includes\(.*\:builds yields no results.
@mbobin this is a good question, and I see that there is a following error present:
ArgumentError: The association scope 'builds' is instance dependent (the scope block takes an argument). Preloading instance dependent scopes is not supported.
I guess that it means that when we extend has_many association with a scope that takes an argument (here: ->(pipeline) { ... }) we can not support preloads, because obviously there may be many pipelines selected in the where. Example: Ci::Pipeline.preload(:builds).all.to_a.
It means that with partitioning we might need to find a different strategy for preloading resources