FIPS - Maven Package Repo - 422 Unprocessable Entity Error

Summary

When pushing a maven package to the repo using mvn deploy, a 422 Unprocessable Entity Error is received and the package is not added to the package registry.

It appears that the following conditions must be present for the issue to occur:

  1. The GitLab server is running on Kubernetes installed using GitLab Helm Chart
  2. The non-FIPS UBI containers are being used
  3. The Kubernetes nodes for the server cluster are in FIPS mode

The error did not occur with the Debian k8s containers or omnibus running on a FIPS VM.

Steps to reproduce

Full details of environment setup and test:

  1. Create an EC2 instance. AMI: ubuntu-pro-fips-server/images/hvm-ssd/ubuntu-focal-20.04-amd64-pro-fips-server-20220615-7bc828d1-c072-4d33-a989-fbad50380cfb ami-04ebe04990712e030.
  • Instance Size: m5.xlarge 4vcpu, 16 Gib Mem Security Group. Open 22, 80, 443, 6443
  1. Update and configure EC2 instance
  • sudo apt update / upgrade
  • sudo hostnamectl set-hostname example.domain.com
  • sudo reboot now
  1. Set up DNS entries
  • new DNS record (example.domain.com) pointing at ec2 instance
  • set wildcard record (*.example.domain.com) pointing at ec2 instance
  1. Install k3s
  • curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="--disable traefik" sh -s -
  • Get kube config and move to the machine where you have kubectl and helm 3 installed
  • Update host names and/or file permissions for kube config as needed.
  1. Install GitLab Using Helm Chart - Use UBI8 containers for webservice and workhorse
  • kubectl create namespace gitlab
  • helm repo add gitlab https://charts.gitlab.io/
  • helm repo update
  • helm upgrade --install gitlab gitlab/gitlab -n gitlab --set certmanager-issuer.email=email@address.com -f values.yaml
  1. Fix Error: The webservice pod will not start - Error from dependencies init container: fe_sendauth: error sending password authentication
  • Get postgres password from secret
  • kubectl exec -it gitlab-toolbox -n gitlab -- /bin/bash
  • psql -d gitlabhq_production -h gitlab-postgresql -u gitlab
  • SET password_encryption = 'scram-sha-256';
  • ALTER USER "gitlab" with password 'DB_PASSWORD_HERE"
  1. Set up gitlab for test
  • Log in as root with default password
  • Create a group
  • Create a project inside the group - blank with README.MD
  • Go to user settings and create a PAT
  1. Execute Test
  • Create a test maven package
  • Update pom.xml with test server URL and project ID
  • Update settings.xml with PAT
  • mvn deploy

Configuration used

Values.yaml: 
global:
  hosts:
    domain: example.domain.com
    externalIP: 1.1.1.1
    gitlab:
      name: gitlab.example.domain.com
    registry:
      name: registry.example.domain.com
    minio:
      name: minio.example.domain.com
  rails:
    bootsnap:
      enabled: false
  shell:
    # Configure the clone link in the UI to include the high-numbered NodePort
    # value from below (`gitlab.gitlab-shell.service.nodePort`)
    port: 32022
gitlab:
  webservice:
    image:
      tag: v15.5.1-ubi8
    minReplicas: 1
    maxReplicas: 1
    resources:
      limits:
       memory: 2G
      requests:
        cpu: 100m
        memory: 900M
    workhorse:
      tag: v15.5.1-ubi8
      resources:
        limits:
          memory: 100M
        requests:
          cpu: 10m
          memory: 10M
  kas:
    minReplicas: 1
    maxReplicas: 1
  sidekiq:
    minReplicas: 1
    maxReplicas: 1
    resources:
      limits:
        memory: 2G
      requests:
        cpu: 300m
        memory: 1G
  gitlab-shell:
    minReplicas: 1
    maxReplicas: 1
    service:
      type: NodePort
      nodePort: 32022
registry:
  hpa:
    minReplicas: 1
    maxReplicas: 1

Current behavior

[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  3.205 s
[INFO] Finished at: 2022-10-29T20:44:19Z
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-deploy-plugin:2.7:deploy (default-deploy) 
on project maven-test: Failed to deploy artifacts: 
Could not transfer artifact net.test.testing:maven-test:jar:1.8-20221029.204416-1 
from/to maven-test (https://URL/api/v4/projects/2/packages/maven): transfer failed for 
https://URL/api/v4/projects/2/packages/maven/net/testlab/testing/maven-test/1.8-SNAPSHOT/maven-test-1.8-20221029.204416-1.jar, 
status: 422 Unprocessable Entity -> [Help 1]

Expected behavior

Package uploaded to maven repo successfully

Versions

  • Chart: gitlab-6.5.1
  • Platform:
    • Self-hosted: k3s
  • Kubernetes:
    • Client: 1.25.3
    • Server: 1.25.3
  • Helm: v3.10.1

Implementing a fix

See #380559 (comment 1183901982)

Edited by David Fernandez