Investigate recommending/supporting kpt as an alternative to separation of config/manifest projects

Original brain dump was in gitlab-org/gitlab!54538 (comment 514883809):

I am starting to think that it is not valuable to support multiple manifest_projects or even have any concept of a manifest project as distinct from the agent config project for that matter.

I think if we really sat down and thought through the functionality people are going to want/need in order to effectively reference multiple/external manifest projects what we'd end up with is exactly what KPT already does well like:

  • ability to lock the git refs you're pointed to and update them easily
  • easy, arbitrary overrides on any resource within an upstream manifest project
  • setting or overriding a default namespace
  • intelligently merging resources together and preventing conflicts

It's also worth noting that KPT does not require any special metadata in order to reference a git repo as a KPT package. See init is optional in their docs.

I can already forsee problems created by having a single agent be responsible for multiple independent reconciliation loops getting us into trouble. We could sink a lot of time into preventing/warning users from doing the wrong thing and it still be both possible and subtle. If we continue supporting this in this way I think we will get a lot of bug reports that are the result of subtle misconfigurations that we have few practical mechanisms to error on.

It may be worth considering how we could build agent config projects around/on top of KPT and have the agent apply all manifests you are referencing deterministically. We could already boil all currently supported agent config down to a simple kustomization.yaml per agent.

Doing so is well within the spirit of @tkuah's suggestion:

  1. To me, the friendlier Developer experience is to start with the same repo, with extensions to the YAML syntax for fetching manifests from other projects.

It really kind of eliminates the need to authorize the agent to pull from external repos at all. You just manage all that in git via kpt pkg get/update.