Associate agent with certain Flux-related resources
Problem
When there are multiple agents installed in the same cluster and have the Flux module enabled (the default), all these agents will act on all Flux GitRepository and Receiver objects.
This is the case because an agent has currently no (sane) way to identify if a certain GitRepository or Receiver object is related to it - for example if the GitRepository is referencing the agent configuration project or that the Receiver object is referencing a GitRepository object that in turn references the agent configuration project. While is sounds feasible (and it actually would be) it's not enough, because a GitRepository can pull in other GitRepository objects on the same instance which will also be affected by this problem.
Note
When there are multiple agents installed in the same cluster
We recommend using a single agent per cluster, but some use cases are not supported yet primarily because of Support for Kubernetes agents at Organization l... (gitlab-org/gitlab#357516 - closed). Instead of implementing custom solutions that go against our design, we should consider seriously if the blocking use cases can be shipped in reasonable time.
Proposals
(1) Magic, no configuration required
An option we could look into would be to let an agent check if they have access to a certain GitRepository (and transitive Receiver) and act on them. This may work okay with a single agent, but with multiple agents they have to race for it because they work completely independently.
This would also add a lot of complexity on our side, but doesn't require any configuration from the user. However, they won't have the flexibility to control which agent should "own" which Flux resource.
(2) Annotation with agent association
The user may provide an annotation like agent.gitlab.com/id: <agent-id> on GitRepository objects. An agent would only act on the object if it matches the agent id from the annotation. The agent would annotate other objects that it "maintains" (like the Receiver and Secret) with its agent id and do the same filtering on informer events.
This wouldn't be a big effort to implement and maintain, but the user would have to configure the agent id. As a mitigation and to make it easy for "simple" setups, we may just allow any agent to reconcile objects that don't have an agent id annotation. We could even think about disabling this using a configuration flag for a certain agent. This gives a lot of flexibility to the user to control what an agent can do.
(3) We don't, use RBAC
One could argue that this is probably a matter that RBAC should solve - that is, an agent and only one agent should have access to a specific GitRepository. However, in practice people often drop their GitRepository into flux-system or have other special requirements, like what a customer mentioned here).
Thus, this is a matter of if we want to support users like the one mentioned before or not.
(4) We don't, organization / top-level agents incoming
We may work on Support for Kubernetes agents at Organization l... (gitlab-org/gitlab#357516 - closed) instead and advertise our users to not use multiple agents in a single cluster. However, I'm not sure if that's really feasible and it wouldn't solve all use cases.
What other options do we have?