Discuss hybrid development environment deployment options for cloud-enabled GDK

Overview

As part of the evaluation of cloud-enabled development environments with GDK (Epic gitlab-org#17447 (closed)), we need to discuss and evaluate different deployment options for running GitLab in development environments, with the flexibility to mix and match services between edit-mode and production-mode configurations.

Deployment Options

The following deployment options have been identified for consideration:

1. Source-based on bare metal (edit-mode) ✍️

  • Traditional approach with source code directly on bare metal
  • Allows for direct code editing and debugging
  • Familiar workflow for current developers
  • Full edit-mode capabilities across all services

2. Dockerized (edit-mode) ✍️

  • Container-based deployment with development-friendly configuration
  • Combines containerization benefits with edit-mode flexibility
  • Known challenges:
    • Volume mounting performance issues for file watching
    • Debugging complexity with containerized processes
    • Port forwarding and networking complications
    • Asset compilation and hot-reloading challenges
    • IDE integration difficulties

3. Precompiled (production-mode) 🚄

  • Uses precompiled assets and production-ready builds
  • Faster startup times but less flexible for development
  • Production-mode configuration for all services

4. Dockerized (production-mode) 🚄

  • Container-based deployment using Docker
  • Better isolation and consistency across environments
  • Production-mode configuration
  • Improved resource management and portability

5. CNG (production-mode) 🚄

  • Cloud Native GitLab deployment using Kubernetes
  • Helm chart-based deployment
  • Production-mode configuration with cloud-native architecture
  • Scalable and cloud-ready approach

Hybrid Service Configuration

A key consideration is the ability to mix and match services between edit-mode and production-mode within the same development environment:

Example Hybrid Scenarios:

  • Rails application in edit-mode + PostgreSQL/Redis in production-mode containers
  • Core GitLab services in edit-mode + Supporting services (Gitaly, Workhorse) in production-mode

Benefits of Hybrid Approach:

  • Improve stability of and optimize performance for services that don't need frequent changes
  • Maintain edit-mode flexibility for actively developed components
  • Reduce resource usage by running stable services in production-mode
  • Enable gradual migration between deployment strategies

Discussion Points

Performance & Resource Usage (#3056 (closed))

  • How do these options compare in terms of startup time, memory usage, and CPU consumption?
  • What are the performance implications of hybrid configurations?

Developer Experience

  • Which combinations provide the best balance of productivity and ease of use?
  • How do debugging workflows differ across these options?
  • What are the learning curves for each approach?

Technical Challenges

  • Dockerized (edit-mode) specific issues:
    • File system performance with volume mounts
    • Hot-reloading and asset compilation reliability
    • Container networking and service discovery
    • Development tool integration (debuggers, profilers)
  • Hybrid configuration complexity:
    • Service communication between different deployment modes
    • Configuration management across mixed environments
    • Dependency resolution and version compatibility

Maintenance & Operations

  • What are the ongoing maintenance requirements for each approach?
  • How do we handle updates and dependency management?
  • What tooling is needed to support hybrid configurations?

Production Parity

  • How closely does each option mirror our production environment?
  • What are the trade-offs between development convenience and production similarity?

GDK Services Deployment Feasibility Matrix

Feasibility matrix based on analysis of the current GDK codebase and service configurations:

Service Edit Likelihood Source-based (edit-mode) Dockerized (edit-mode) Precompiled (production-mode) Dockerized (production-mode) CNG (production-mode) Notes
Rails Web 🔥 Very High ✅ High ❌ Low ✅ High ✅ High ✅ High Edit-mode Docker has volume mounting performance issues
GitLab Workhorse 🟡 Medium ✅ High ❌ Low ✅ High ✅ High ✅ High Go binary, minimal edit requirements
Webpack 🟡 Medium ⚠️ Medium ❌ Low ✅ High ✅ High ✅ High Third-party build tool, only config changes. Docker volume issues affect performance
Vite 🟡 Medium ⚠️ Medium ❌ Low ✅ High ✅ High ✅ High Third-party build tool, only config changes. HMR problematic in containers
Gitaly 🟢 Low ✅ High ❌ Low ✅ High ✅ High ✅ High Go service, socket communication may need adjustment
Praefect 🟢 Low ✅ High ❌ Low ✅ High ✅ High ✅ High Git proxy service, similar to Gitaly
GitLab Shell 🟢 Low ✅ High ❌ Low ✅ High ✅ High ✅ High SSH access may require special container setup
GitLab AI Gateway 🟡 Medium ✅ High ❌ Low ✅ High ✅ High ✅ High Python/Go service, newer component
Duo Workflow Service 🟡 Medium ✅ High ❌ Low ✅ High ✅ High ✅ High AI workflow orchestration, newer component
GitLab Pages 🟢 Low ✅ High ❌ Low ✅ High ✅ High ✅ High Static site serving
Container Registry 🟢 Low ✅ High ❌ Low ✅ High ✅ High ✅ High Docker registry service
GitLab Runner 🟢 Low ✅ High ❌ Low ✅ High ✅ High ✅ High CI/CD executor
Third-party dependencies
PostgreSQL ⚪ Never n/a ❌ Low ✅ High ✅ High ✅ High Database operations work well in all modes
Redis ⚪ Never n/a ❌ Low ✅ High ✅ High ✅ High In-memory store, no file watching needed
Prometheus ⚪ Never n/a ❌ Low ✅ High ✅ High ✅ High Metrics collection
Grafana ⚪ Never n/a ❌ Low ✅ High ✅ High ✅ High Metrics visualization
SSHD ⚪ Never n/a ❌ Low ✅ High ✅ High ✅ High SSH daemon needs privileged container access
Nginx ⚪ Never n/a ❌ Low ✅ High ✅ High ✅ High Reverse proxy, configuration-driven

Legend:

  • ✅ High: Excellent feasibility, minimal issues expected
  • ⚠️ Medium: Feasible with some challenges or limitations
  • ❌ Low: Significant challenges, not recommended

Edit Likelihood:

  • 🔥 Very High: Core application code, frequently modified during development
  • 🔶 High: GitLab services actively developed and maintained by GitLab teams
  • 🟡 Medium: GitLab services with moderate development activity
  • 🟢 Low: Configuration changes only, source rarely modified
  • ⚪ Never: Third-party dependencies, never edited during GitLab development

Key Insights:

Optimal Hybrid Strategy by Edit Likelihood:

  1. Very High Edit (🔥): Keep in edit-mode for maximum development flexibility
  2. High Edit (🔶): Edit-mode preferred when actively developing, production-mode acceptable otherwise
  3. Medium Edit (🟡): Production-mode containers recommended, edit-mode when actively developing
  4. Never Edit (⚪): Strong candidates for production-mode containers or CNG deployment

Dockerized edit-mode is not recommended (❌ Low) due to unresolved Developer Experience issues.

Conclusion

After interviewing @andrewn, we gained consensus on a hybrid approach that prioritizes developer productivity over using CNG or Kubernetes for development.

Dockerized edit-mode is not recommended due to unresolved Developer Experience issues.

For now, we strongly recommend source-based development for edit-mode services and containerization for use-only services.

Using other approaches (Dockerized edit-mode or CNG) will still be possible after resolving their outstanding issues.

Next: Modularize services and start isolating them.

Edited by Peter Leitzen