Move ai-assist Kubernetes manifests to CI
The manifests in https://gitlab.com/gitlab-org/modelops/applied-ml/code-suggestions/ai-assist are right now applied manually from the workstation. We should try to move this to CI so that changes can be tracked and audited.
At this time, we don't want to completely re-write the configuration or adopt it into existing repositories. We think as a small iteration we can convert what we have into Helm and use the existing project's CI to apply it.
From gitlab-org/modelops/applied-ml/code-suggestions/ai-assist#108 (comment 1401281211)
Proposed Steps for IAC on the Code Suggestions Project
-
Rewrite the manifest files in helm - Manifest files currently reside in https://gitlab.com/gitlab-org/modelops/applied-ml/code-suggestions/ai-assist/-/tree/main/manifests
- A lot of this migration work has already been done in gitlab-org/modelops/applied-ml/code-suggestions/ai-assist!85 (merged).
- It would be good to identify exactly which resources in https://gitlab.com/gitlab-org/modelops/applied-ml/code-suggestions/ai-assist/-/tree/main/manifests are being used with production code suggestions right now and focus on those first.
-
Migrate staging and production from the manifests to helm - Import the existing manifests into Helm by applying the correct helm annotations to the existing manifests.
- This is not something I've done before, but reading https://jacky-jiang.medium.com/import-existing-resources-in-helm-3-e27db11fd467 seems to indicate that it is possible.
-
render.sh
has been written to help with this. It uses helm to rewrite the existing manifests, but adds additional annotations as expected by helm. - Apply the manifests one final time with
kubectl apply
to add the annotations. - Helm should now recognise the resources as helm managed.
- Run
helm upgrade
to apply any changes.
- Import the existing manifests into Helm by applying the correct helm annotations to the existing manifests.
-
Remove the manifest files completely and start relying on manual helm upgrades to apply changes. -
Automate the deployment of the project - Use the existing project: https://ops.gitlab.net/gitlab-com/gl-infra/config-mgmt/-/tree/master/environments/ai-assisted-legacy-prd
- Configure ip access for deployment runners running on ops mirror (same)
- Provide access for the team to the ops mirror for the https://ops.gitlab.net/gitlab-com/gl-infra/config-mgmt project
- Install the GitLab/GCP OIDC Module for establishing a trust relationship between the
ai-assist
project and the GCP project. This can be done with the Terraform module: https://gitlab.com/gitlab-com/gl-security/security-operations/infrastructure-security-public/oidc-modules -- I have some POC Terraform/Script to help with this step and get whoever needs to do this going - Write a CI/CD script for automatically performing a
helm diff
on merge request branch pipelines (same) - Add a CI/CD job for automatically applying a
helm upgrade
on pipelines from themain
branch. (same)
At this point, we should have fully automated IaC deployment for Code Suggestions and can move forward with incremental changes.
@andrewn has created a PoC the Helm conversion in gitlab-org/modelops/applied-ml/code-suggestions/ai-assist!85 (merged)