Add custom certificates support [Attempt 3] (!134) · Merge requests · GitLab.org / Cloud Native / charts / gitlab-zoekt

What does this MR do and why?

This merge request updates the GitLab Zoekt helm chart from version 3.5.1 to 3.5.2, adding flexible container customization capabilities with a critical fix for the certificates.initContainers.enabled flag.

Note

This is the third attempt after fixing the bug discovered during the revert of MR !131. The issue was tracked in gitlab-com/gl-infra/delivery#21528 (comment 2802994721).

The main enhancement allows users to add their own custom containers, storage volumes, and initialization scripts to the deployment. This includes support for:

Extra init containers - containers that run before the main application starts (useful for setup tasks like updating security certificates)
Extra volumes - additional storage that can be mounted from secrets or configuration files
Extra volume mounts - ways to attach the extra storage to specific locations in containers
Extra containers - additional containers that run alongside the main application (like log shipping or monitoring tools)

The changes include comprehensive test coverage that verifies these new features work correctly both individually and when combined together. The tests cover scenarios like adding custom certificate authority updates, configuration management, and log shipping containers.

The implementation follows GitLab's standard patterns for container customization, making it consistent with other GitLab components. This gives users much more flexibility to customize their Zoekt search deployments for specific security, monitoring, or operational requirements without modifying the core chart templates.

Issues addressed:

Critical Bug Fix: `certificates.initContainers.enabled` Flag

⚠️ What Went Wrong in MR !131 (merged)

After the initial implementation (MR !131 (merged)), we discovered that setting certificates.initContainers.enabled: false didn't actually disable certificate init containers. This caused severe performance issues in GitLab.com deployments:

Problem: Deployment times increased from ~15 minutes to 40+ minutes
Root cause: Certificate init containers were being added to all 36 Zoekt pods during rolling updates, even though Zoekt doesn't need Gitaly/Praefect TLS certificates
Impact: The majority of the total deployment time was spent on Zoekt pod rolling updates

Root Cause Analysis

The bug was caused by the default function behavior. According to the documentation, the default function treats false as an "empty" value:

Broken code (MR !131 (merged)):

{{- $certificateInitContainersEnabled := .Values.certificates.initContainers.enabled | default true -}}

When GitLab.com set:

gitlab-zoekt:
  certificates:
    initContainers:
      enabled: false

The template evaluated:

false | default true  →  true  ❌

The false value was treated as "empty" and replaced with the default true, so the flag didn't work!

The Fix

Fixed code (this MR):

{{- $certificateInitContainersEnabled := true -}}
{{- if hasKey .Values.certificates.initContainers "enabled" -}}
  {{- $certificateInitContainersEnabled = .Values.certificates.initContainers.enabled -}}
{{- end -}}

This properly handles all three cases:

enabled: false → $certificateInitContainersEnabled = false ✅
enabled: true → $certificateInitContainersEnabled = true ✅
Not specified → $certificateInitContainersEnabled = true (default) ✅

Why Tests Didn't Catch It

The existing tests passed even with the bug because:

Tests run in standalone mode where gitlab.certificates.initContainer is an empty stub template
The bug only manifests when used as a GitLab subchart, where that template is overridden with actual certificate init container logic
So even when $hasCertificates was incorrectly true, the output was still empty in tests

This bug required integration testing in the full GitLab chart to detect.

Test Improvements

Added additional test coverage to verify the initContainers section behavior:

it 'verifies the template logic by checking initContainers section is truly absent' do
  t = HelmTemplate.new(values)
  pod_spec = t.dig(path, 'spec', 'template', 'spec')
  expect(pod_spec['initContainers']).to be_nil
end

While these tests still can't fully catch the subchart integration issue, they provide better coverage for the standalone behavior.

Performance Fix Validation

Template Verification

# Test with enabled: false
helm template test . -f tmp/test-cert-disabled.yaml | grep -A 10 "initContainers:"
# Result: No initContainers found ✅

# Test with enabled: true
helm template test . -f tmp/test-cert-enabled.yaml
# Result: Renders successfully ✅

Expected Performance Impact

Once deployed to GitLab.com with certificates.initContainers.enabled: false:

Deployment time reduction: ~25 minutes (from 40+ back to ~15 minutes)
Rolling update optimization: Certificate processing eliminated from all 36 Zoekt pods
Resource efficiency: Reduced container startup overhead during deployments

How to set up and validate locally

1. Basic Template Validation ✅ VERIFIED

helm template test . -f tmp/test-basic-values.yaml
helm lint -f tmp/test-basic-values.yaml

Results:

✅ All templates render correctly
✅ No linting errors (only expected warnings for missing values)

2. Certificate Flag Test (Disabled) ✅ VERIFIED

helm template test . -f tmp/test-cert-disabled.yaml

Configuration:

certificates:
  initContainers:
    enabled: false
global:
  certificates:
    customCAs:
      - secret: company-internal-ca
        keys: [company-ca.crt]

Results:

✅ No initContainers section rendered (correct behavior)
✅ Certificate init containers properly disabled
✅ Flag works as expected

3. Certificate Flag Test (Enabled) ✅ VERIFIED

helm template test . -f tmp/test-cert-enabled.yaml

Configuration:

certificates:
  initContainers:
    enabled: true
global:
  certificates:
    customCAs:
      - secret: company-internal-ca
        keys: [company-ca.crt]

Results:

✅ Templates render correctly
✅ Ready for GitLab subchart certificate integration

4. GitLab Chart Integration Test

Test Command:

cd /path/to/gitlab-chart
helm template gitlab . \
  --set gitlab-zoekt.install=true \
  --set gitlab-zoekt.certificates.initContainers.enabled=false \
  --set global.certificates.customCAs[0].secret=company-ca

Expected Results:

✅ No certificate init containers in gitlab-zoekt pods
✅ Flag properly prevents certificate processing
✅ Deployment performance improved

5. Run Tests

bundle exec rspec

Results:

✅ 117 examples, 0 failures

Technical Details

New Values Configuration

certificates:
  initContainers:
    enabled: true  # Default: true for backwards compatibility

# Extra configurations using GitLab chart patterns
extraInitContainers: ""
extraVolumes: ""
extraVolumeMounts: ""
extraContainers: ""

# Init container configuration (required for GitLab certificate integration)
init:
  image: {}
  resources:
    requests:
      cpu: 50m
  containerSecurityContext:
    runAsUser: 1000
    allowPrivilegeEscalation: false
    runAsNonRoot: true
    capabilities:
      drop: [ "ALL" ]

Template Override Architecture

When used as GitLab subchart:

Main chart templates (gitlab/templates/_certificates.tpl) override our stub templates
Global certificate configuration automatically flows to all subcharts
Manual escape hatch still available via extraInitContainers patterns
Performance flag (certificates.initContainers.enabled) now works correctly

Reusable Template Helper

Centralized logic: Single source of truth for certificate and extra init container conditions
Nil-safe: Proper defensive checking prevents template rendering failures
Bug-fixed: Uses hasKey instead of default to properly handle false values
Flexible: Works for both deployment and statefulset with different additional requirements
Maintainable: Changes to common logic only need to be made in one place

Expected Results

✅ Templates render correctly: No more nil pointer errors or template failures
✅ Dual compatibility: Works standalone AND as GitLab subchart seamlessly
✅ Certificate flag works: enabled: false properly prevents certificate init containers
✅ Performance improvement: 25-minute reduction in GitLab.com deployment times
✅ Certificate Integration:
- Standalone: Graceful degradation with empty certificate helpers
- GitLab subchart: Full certificate processing via template override (when enabled)
- Manual: Escape hatch via extraInitContainers for standalone custom CAs
✅ All tests pass: 117 examples, 0 failures
✅ Zero Regressions: All existing functionality preserved

Migration Notes

No Breaking Changes:

All existing functionality preserved
No changes to values.yaml structure required (except version bump)
Same template rendering results for existing configurations
Additional certificate integration capabilities when used as GitLab subchart

Backwards Compatibility:

Standalone deployments continue to work exactly as before
GitLab subchart integration enhanced but not changed
Manual certificate injection via extraInitContainers unchanged
certificates.initContainers.enabled defaults to true (enabled)

For GitLab.com: To enable the performance fix, set in your values:

gitlab-zoekt:
  certificates:
    initContainers:
      enabled: false

This will reduce Zoekt deployment times from 40+ minutes back to ~15 minutes.

Changelog

Fix certificates.initContainers.enabled flag using hasKey instead of default
Add comprehensive test coverage for certificate flag behavior
Bump version from 3.4.0 to 3.5.2
Add support for extra init containers, volumes, volume mounts, and containers
Add GitLab-compatible certificate template stubs

Edited Oct 07, 2025 by Dmitry Gruzd

Add custom certificates support [Attempt 3]

What does this MR do and why?

Critical Bug Fix: certificates.initContainers.enabled Flag

Root Cause Analysis

The Fix

Why Tests Didn't Catch It

Test Improvements

Performance Fix Validation

Template Verification

Expected Performance Impact

How to set up and validate locally

1. Basic Template Validation ✅ VERIFIED

2. Certificate Flag Test (Disabled) ✅ VERIFIED

3. Certificate Flag Test (Enabled) ✅ VERIFIED

4. GitLab Chart Integration Test

5. Run Tests

Technical Details

New Values Configuration

Template Override Architecture

Reusable Template Helper

Expected Results

Migration Notes

Changelog

Merge request reports

Critical Bug Fix: `certificates.initContainers.enabled` Flag