Explore paths to make the usage of CNG images feasible for cloud-enabled DevEnv

Summary

This issue explores the feasibility of using CNG (Cloud Native GitLab) images as part of the broader effort to evaluate Cloud-enabled GDK for remote development.

Context

Currently, GDK has 52 services and uses precompiled binaries that are built separately from production. This creates potential inconsistencies between development and production environments and duplicates build effort.

Key question around versioning

Based on discussion with the Delivery team:

Image availability

• Release versions: CNG images are available for stable release versions (e.g., registry.gitlab.com/gitlab-org/build/cng/gitlab-gitaly:v18.4.0)
• Development versions: Images for specific commit SHAs (like those in GITALY_SERVER_VERSION) are built for auto-deploy but stored on the dev instance
• Community access: Dev instance images aren't accessible to community contributors or customers

Current workflow challenges

1. Version mapping: No straightforward way to map GITALY_SERVER_VERSION commit SHA to CNG image tag
2. Development vs production: Using release versions loses ability to reproduce bugs in specific commit versions
3. Access restrictions: Auto-deploy images with exact commit versions are only available on dev instance

Potential solutions

1. For stable branches: Use corresponding release version images
2. For master/feature branches:
   • Accept using latest release version (loses exact version matching)
   • Build images locally (duplicates effort)
   • Mirror auto-deploy images to .com (security concerns with pre-release fixes)

Services not included in CNG

AI Gateway

Based on discussion with the AI Gateway team, AI Gateway is not currently part of CNG images or charts.gitlab.io because:

• It was developed independently without CNG integration support
• Uses custom helm chart registry instead of charts.gitlab.io
• There may be future work to integrate it for Dedicated deployments (see https://gitlab.com/groups/gitlab-org/-/epics/18775)

Other Affected Services

Other newer or niche services are likely also affected, particularly:

• Runway-hosted services: Services deployed via Runway instead of CNG/Helm charts
• 3rd-party services: External dependencies like OpenBao
• Newer services: Recently added services that haven't been integrated into CNG yet

This significantly impacts the feasibility of using CNG for complete local development environments, as key services would still require separate setup.

Next Steps

• Consider building local images for exact version matching (performance intensive!)
• Explore hybrid approach using release images for most cases, local builds for specific needs
• Audit all GDK services to identify which are/aren't available in CNG
• Investigate integration path for non-CNG services (AI Gateway, Runway services, etc.)
• Evaluate impact of missing services on cloud-enabled GDK viability

Related

Epic: gitlab-org&17447