Encryption key rotation

Rémy Coutablerequested to merge
rymai-main-patch-b1c7 into main

Review app: https://gitlab-com.gitlab.io/content-sites/handbook/mr10303/handbook/engineering/architecture/design-documents/encryption_keys_rotation/

Introduction

We need a solution to rotate GitLab's encryption keys (used to protect sensitive data at rest) without requiring system downtime, addressing a significant security and operational risk for both our customers and GitLab.com. The proposed solution enables zero-downtime key rotation through support for multiple concurrent keys, automated background re-encryption, and a deployment workflow optimized for multi-node installations, allowing organizations to maintain security best practices without service interruption.

Objectives

This design document addresses a big inconvenience in GitLab's current encryption key management system. Currently, GitLab's encryption keys, which protect sensitive data at rest in the database, cannot be rotated without taking the entire system offline. This limitation poses significant business risks:

  1. In the event of a key compromise, customers would face service disruption during key rotation
  2. For GitLab.com and large enterprise deployments, the required downtime makes regular key rotation practically impossible
  3. With the expansion of GitLab's Dedicated and Cells infrastructure, the risk of key compromise increases, as does the operational complexity of key management across multiple instances

The proposed solution enables zero-downtime encryption key rotation through:

  • Support for multiple concurrent encryption keys
  • Automated background re-encryption of data
  • Clear visibility into key usage and rotation progress through a new admin interface
  • A deployment workflow optimized for multi-node installations

Key business benefits:

  • Eliminates service disruption during security-critical key rotations
  • Enables compliance with security policies requiring regular key rotation
  • Reduces operational risk in multi-instance deployments, especially in the context of Cells
  • Provides foundation for secure data movement between GitLab instances, especially when using the Cells Org Mover

The implementation is planned across seven iterations, focusing on maintaining system stability while introducing this capability. This project directly supports GitLab's scaling initiatives and enterprise security requirements.

Key Changes

Area Change
Architecture Introduces support for multiple encryption keys
Security Enables online rotation of encryption secrets
Monitoring New admin interface for tracking encryption status
Performance Background re-encryption process with throttling
Compatibility Supports existing encryption frameworks: ActiveRecord::Encryption, attr_encrypted, and TokenAuthenticatable

Next Steps

  1. Review the technical approach and architecture decisions
  2. Validate the proposed background re-encryption process
  3. Assess impacts on system performance
  4. Plan phased implementation timeline

Approvals

References

Edited by Krasimir Angelov

Merge request reports