Deployment workflow management with first-class Service concept

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

Release notes

Problem to solve

As a lead engineer / platform engineer, I want a single place to define the lifecycle and related processes of a service, so GitLab can support me in Day 0, Day 1 and Day 2 operations.

During the CMS for Environment Management, the struggles with the current promote/rollback feature kept coming up. (source)

Deployment rollback feature is too straightforward to be used in a real-world complicated usecases. (example)

Use cases

  • As a Platform engineer, I want a single view to see what's running in my environments, across environments.
  • As a Platform engineer, I want a single view to see what's running in a specific environment.
  • As a Application Operator, I want a single view to see what version of my services is running in a specific environment.
  • As a Application Operator, I want a single view to see what version of my services is running across environments.
  • As a Development Team Lead, I want a single view to see how many environments are running, and be able to stop/start them to minimize costs on ephemeral environments.
  • As a Platform engineer, I want to know what services/environments might need any action from.
  • As a Platform engineer, I want to define environments in a single place and share them across the organization for deploying services to them.
  • As a Platform engineer, I want to define automated processes (optionally with manual guardrails) to enrol a service into an environment.
  • As a Platform engineer, I want to define automated processes (optionally with manual guardrails) to deploy a new version of a service to an environment.

Intended users

User experience goal

  1. We want to build a two-dimensional system of services and environments
dev staging prod
fancy API 1.2-alex; 1.2-shinya 1.1 1.0
KAS 16.1-rc2 16.1-rc1 16.0
  1. The entry point is the service
  2. Services are defined at the project level
  3. Each project might have 1-N services
  4. A deployment of a service can be promoted between environments (promote fancy API 1.1 from staging to prod)
  5. Define entities and processes as code

Also:

  • Users can see the list of services in service index page.
  • Users can see the details of services in service details page.

Proposal

TBD. Here are the rough proposal.

(The proposal builds on the ideas presented by @codablock in #357947 (comment 1194665028) and separate calls)

Define a .gitlab/services.yml file:

services:
- name: fancy_api
  match_labels:
    app.gitlab.com/name: fancy_api
  environments:
  - name: dev
    deploy_on:
      automatically: on_commit
      branch: dev/*
    prepare_environment: .gitlab/services/prepare-dev.yml
    deploy_environment: .gitlab/services/deploy-dev.yml
    cleanup_environment: .gitlab/services/review-cleanup.yml
  - name: staging
    deploy_on:
      automatically: on_release
      release_pattern: fancy_api-{semver}
    prepare_environment: .gitlab/services/prepare-staging.yml
    deploy_environment: .gitlab/services/deploy-staging.yml
  - name: prod
    deploy_on:
      release_pattern: fancy_api-{semver}
      approvals:
      - group: tech leads
      - group: pm
      deploy:
      - group: release managers
      check_artifacts: .gitlab/services/validate-prod.yml
    prepare_environment: .gitlab/services/prepare-prod.yml
    deploy_environment: .gitlab/services/deploy-prod.yml
- name: kas
  ...

The core ideas are to:

  • define the services in code
  • each service has an (ordered) list of environments it deploys to
  • the environments can be deployed automatically when conditions are met (new artifact with given pattern exists) or manually
  • the environment deployments have the input artifact defined (it can be a branch commit or a release artifact)
  • all the processes are defined in separate GitLab pipeline definition files (or something similar)
  • we need a mechanism to retrieve service specific metrics from a metrics server (match_labels was used for this here)

Non goals

  • we are not focusing on showing metrics; this is the domain of groupobservability and we can move forward by connecting existing data to the discussed concepts
  • we are not focusing on registering services as code (see: rough proposal above); while we might want to enable code-based service registration, a lot of it can be achieved if we provide an API and a Terraform resource to manage the services through code

Further details

  • This issue defines conceptual design and vision. It's likely a few years of work to complete.

Permissions and Security

Documentation

Availability & Testing

Available Tier

Feature Usage Metrics

What does success look like, and how can we measure that?

What is the type of buyer?

Is this a cross-stage feature?

What is the competitive advantage or differentiation for this feature?

Customer requests / workaround

  • I want to see more details about the environment status, than the last pipeline (source)
  • I want to manage resources at the service level (source: Design Sprint)
  • Very interested in the mock-ups (GitLab, Scalability)
  • Currently, Delivery, Scalability, and SREs are working around with service-catalog.yml. See this comment for more info. Process doc
  • Delivery uses a project on OPS, named release-metadata, as a ledger of every package we produce, either auto_deploy or on-premise release. https://ops.gitlab.net/gitlab-org/release/metadata (source)
  • Runway Service manifest schema

Deprecations

Following features are deprecated in favor of this new feature:

  • ...

This page may contain information related to upcoming products, features and functionality. It is important to note that the information presented is for informational purposes only, so please do not rely on the information for purchasing or planning purposes. Just like with all projects, the items mentioned on the page are subject to change or delay, and the development, release, and timing of any products, features, or functionality remain at the sole discretion of GitLab Inc.

Edited by 🤖 GitLab Bot 🤖