Summary

Add protobuf-first configuration management module for LabKit v2 (v2/config) that provides standardized configuration loading, validation, and migration capabilities across GitLab's Go services.

Related to gitlab#591894

Features

Core Capabilities

Format autodetection: YAML, JSON (always available), TOML (opt-in via WithParser)
Strong validation: Uses protovalidate with CEL expressions for rich constraint validation
Typed generic migrations: func(source S) (T, error) - type-safe transformations with compiler checks
Pre/post-migration validation: Validates against both v(N-1) and v(N) schemas
Rollback safety: Unknown fields ignored by default (forwards-compatible)
Strict mode: Optional for CI validation to catch typos and obsolete config
Extensible: Add custom format parsers via simple Parser interface

Protocol Buffers Edition 2023

All proto files use modern edition = "2023" syntax instead of syntax = "proto3":

Future-proof approach aligned with protobuf roadmap
Explicit feature declarations (features.field_presence = IMPLICIT)
Easy migration path to Edition 2024+ when buf supports it
Wire format remains identical (fully backward compatible)

Typed Migration System

func migrateV1ToV2(source *configv1.Config) (*configv2.Config, error) {
    return &configv2.Config{
        Version: 2,
        Server: &configv2.ServerConfig{
            Address: source.Server.Host,  // Type-safe! IDE autocomplete!
            Port:    source.Server.Port,
            Timeout: durationpb.New(30 * time.Second),
        },
        Logging: source.Logging,
    }, nil
}

loader, _ := config.New(config.WithMigration(migrateV1ToV2))

Benefits:

Type safety with compiler checks
IDE autocomplete and refactoring support
Easier to write and test
No type assertions or map manipulation
Only a single major version upgrade is supported by a service at a time. This is an explicit guardrail - by design, so that service teams are required to clean-up old configurations (<n-1) instead of letting them accrue in the application. Given that customers cannot skip over more than one major version upgrade, this is a reasonable limitation.

Review Points

Double Validation

Pre-migration: Validate against v(N-1) schema
Migration: Transform typed source to typed target
Post-migration: Validate against v(N) schema

Examples

Standard Usage

	loader, err := config.New()
	if err != nil {
		log.Fatal(err)
	}

	var cfg examplev2.Config
	if err := loader.Load("donkey.yaml", &cfg); err != nil {
		log.Fatal(err)
	}

	...

Migration


func migrateV1ToV2(source *configv1.Config) (*configv2.Config, error) {
    return &configv2.Config{
        Version: 2,
        Server: &configv2.ServerConfig{
            Address: source.Server.Host,  // Type-safe! IDE autocomplete!
            Port:    source.Server.Port,
            Timeout: durationpb.New(30 * time.Second),
        },
        Logging: source.Logging,
    }, nil
}

func main() {
	loader, err := config.New(config.WithMigration(migrateV1ToV2))
	if err != nil {
		log.Fatal(err)
	}

	var cfg examplev2.Config
	if err := loader.Load("donkey.yaml", &cfg); err != nil {
		log.Fatal(err)
	}

	...

Documentation

LabKit v2 Config

Protobuf-first configuration management for GitLab Go services.

Overview

The config package provides a standardized way to load, validate, and migrate configuration files using Protocol Buffers as the single source of truth for configuration schemas. Validation rules are defined using protovalidate constraints directly in .proto files.

Features

Format autodetection: Automatically detects YAML, JSON, or TOML by file extension
Strong validation: Uses protovalidate for rich constraint validation (ranges, patterns, cross-field rules)
Rollback safety: Unknown fields silently ignored by default (forwards-compatible)
Version migrations: Automatic migration from version N-1 to N with pre/post-validation
Excellent error messages: File, line, column information for parse errors (YAML)
Extensible: Add custom format parsers via simple interface

Installation

# Install buf (if not using mise/asdf)
# See: https://buf.build/docs/installation

# Or with mise/asdf (recommended)
mise install

Quick Start

1. Define your configuration proto

// proto/config/v1/config.proto
edition = "2023";
package myapp.config.v1;

option go_package = "myapp/gen/config/v1;configv1";
option features.field_presence = IMPLICIT;

import "buf/validate/validate.proto";

message Config {
  int32 version = 1;
  ServerConfig server = 2 [(buf.validate.field).required = true];
}

message ServerConfig {
  string host = 1 [(buf.validate.field).string.min_len = 1];
  uint32 port = 2 [(buf.validate.field).uint32 = {
    gte: 1
    lte: 65535
  }];
}

2. Generate Go code

# Using buf directly (from v2/config directory)
cd v2/config
buf generate

# Or use the provided script for config module + examples (from repo root)
./scripts/generate-config-protos.sh

3. Load and validate configuration

package main

import (
    "log"
    
    "gitlab.com/gitlab-org/labkit/v2/config"
    configv1 "myapp/gen/config/v1"
)

func main() {
    loader, err := config.New()
    if err != nil {
        log.Fatal(err)
    }
    
    var cfg configv1.Config
    if err := loader.Load("config.yaml", &cfg); err != nil {
        log.Fatal(err)
    }
    
    // cfg is now loaded and validated
    log.Printf("Server: %s:%d", cfg.Server.Host, cfg.Server.Port)
}

Format Support

Always Available

YAML (.yaml, .yml) - Recommended for human-authored configs
JSON (.json) - Useful for machine-generated configs

Opt-in

TOML (.toml) - Add via WithParser(toml.NewTOMLParser())

import (
    "gitlab.com/gitlab-org/labkit/v2/config"
    "gitlab.com/gitlab-org/labkit/v2/config/toml"
)

loader, _ := config.New(
    config.WithParser(toml.NewTOMLParser()),
)

Validation

Validation uses protovalidate constraints defined inline in proto files.

Standard Constraints

message ServerConfig {
  string host = 1 [(buf.validate.field).string = {
    min_len: 1
    max_len: 255
  }];
  
  uint32 port = 2 [(buf.validate.field).uint32 = {
    gte: 1
    lte: 65535
  }];
  
  repeated string tags = 3 [(buf.validate.field).repeated.min_items = 1];
}

Cross-Field Validation with CEL

message TLSConfig {
  string cert_path = 1;
  string key_path = 2;
  
  option (buf.validate.message).cel = {
    id: "tls_pair"
    message: "cert_path and key_path must both be set or both be empty"
    expression: "(this.cert_path == '') == (this.key_path == '')"
  };
}

Migrations

Configure version migrations to automatically upgrade configs from version N-1 to N.

1. Define versioned protos

v1 config:

message Config {
  int32 version = 1;
  ServerConfig server = 2;
}

message ServerConfig {
  string host = 1;
  uint32 port = 2;
}

v2 config (renamed field):

message Config {
  int32 version = 1;
  ServerConfig server = 2;
}

message ServerConfig {
  string address = 1;  // renamed from "host"
  uint32 port = 2;
}

2. Write typed migration function

func migrateV1ToV2(source *configv1.Config) (*configv2.Config, error) {
    return &configv2.Config{
        Version: 2,
        Server: &configv2.ServerConfig{
            Address: source.Server.Host, // Renamed field
            Port:    source.Server.Port,
        },
    }, nil
}

3. Register migration

loader, _ := config.New(
    config.WithMigration(migrateV1ToV2),
)

var cfg configv2.Config
// If config file is v1, migration runs automatically
if err := loader.Load("config.yaml", &cfg); err != nil {
    log.Fatal(err)
}

Migration Flow

Parse config file into target proto (v2) to detect version
Detect version mismatch (e.g., file has v1, proto expects v2)
Re-parse config file into source type (v1)
Pre-migration validation: Validate against v1 schema
Run typed migration function to transform v1 → v2
Post-migration validation: Validate against v2 schema

The migration function receives fully-typed, validated v1 config and returns v2 config. This provides type safety and makes migrations easier to write and test.

Strict Mode

By default, unknown fields are silently ignored (rollback-safe). Enable strict mode to catch typos:

loader, _ := config.New(config.WithStrictMode())

When to use strict mode

✓ Use in:

CI pipelines validating config examples
Pre-deployment validation
Development environments

✗ Don't use in:

Production deployments (breaks rollback safety)
Canary deployments (version skew)

Why rollback safety matters

If you deploy v2 with a new config field, then roll back to v1, the v1 binary will encounter an unknown field. With strict mode enabled, it crashes. With strict mode disabled (default), it ignores the field and continues.

Custom Parsers

Add support for custom formats by implementing the Parser interface:

type Parser interface {
    Extensions() []string
    Unmarshal(data []byte, path string, msg proto.Message, strict bool) error
}

Example: XML parser

type xmlParser struct{}

func (p *xmlParser) Extensions() []string {
    return []string{".xml"}
}

func (p *xmlParser) Unmarshal(data []byte, path string, msg proto.Message, strict bool) error {
    // Parse XML, convert to proto message
    return nil
}

// Register the parser
loader, _ := config.New(
    config.WithParser(&xmlParser{}),
)

API Reference

Constructors

// Create loader with default settings (YAML + JSON parsers)
func New(opts ...Option) (*Loader, error)

Options

// Add custom format parser
func WithParser(p Parser) Option

// Enable strict mode (reject unknown fields)
func WithStrictMode() Option

// Register migration from v(N-1) to v(N)
func WithMigration(fn MigrationFunc, oldProto proto.Message) Option

Methods

// Load config from file (format auto-detected)
func (l *Loader) Load(path string, msg proto.Message) error

// Load config from bytes with explicit format
func (l *Loader) LoadBytes(data []byte, ext string, msg proto.Message) error

Helpers

// Extract version field from proto message
func GetVersion(msg proto.Message) (int, error)

Error Handling

Parse Errors (YAML)

config.yaml:5:3: invalid ServerConfig.port: value must be <= 65535 but got 99999

Validation Errors

validation failed: invalid Config.server: embedded message failed validation |
  caused by: invalid ServerConfig.port: value must be greater than or equal to 1

Migration Errors

migration from version 1 to 2 failed: server.host field is required

Best Practices

Use YAML for human-authored configs - Better readability, comments supported
Use JSON for machine-generated configs - Simpler, no ambiguity
Keep migrations simple - Only support N-1 → N, not arbitrary jumps
Use protovalidate for all constraints - Single source of truth
Test config examples in CI with strict mode - Catch typos early
Never use strict mode in production - Breaks rollback safety

Dependencies

buf.build/go/protovalidate - Validation engine
buf.build/go/protoyaml - YAML parser with rich errors
google.golang.org/protobuf - Protocol Buffers runtime
github.com/pelletier/go-toml/v2 - TOML parser (opt-in)

Example

See example/ directory for a complete working example with:

Versioned proto definitions (v1 and v2)
Migration function
Example config files
Main program demonstrating all features

cc @WarheadsSE

Edited Mar 05, 2026 by Andrew Newdigate

feat(v2/config): Add protobuf-first configuration management with typed migrations

Summary

Features

Core Capabilities

Protocol Buffers Edition 2023

Typed Migration System

Review Points

Double Validation

Examples

Standard Usage

Migration

Documentation

LabKit v2 Config

Overview

Features

Installation

Quick Start

1. Define your configuration proto

2. Generate Go code

3. Load and validate configuration

Format Support

Always Available

Opt-in

Validation

Standard Constraints

Cross-Field Validation with CEL

Migrations

1. Define versioned protos

2. Write typed migration function

3. Register migration

Migration Flow

Strict Mode

When to use strict mode

Why rollback safety matters

Custom Parsers

API Reference

Constructors

Options

Methods

Helpers

Error Handling

Parse Errors (YAML)

Validation Errors

Migration Errors

Best Practices

Dependencies

Example

Related Documentation

Merge request reports