FIPS errors after upgrading to v18.2.x
Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.
Summary
Upgrading GitLab-ee FIPS to version 18.2.x results in FIPS related failures in the logs. One symptom is noticing NGINX load balancer handshake errors along with FIPS related error messages in the praefect logs. Another is noticing 502 errors in the GitLab pages logs (/var/log/gitlab/gitlab-pages/current).
Steps to reproduce
- Have FIPS enabled on Gitaly cluster and Praefect cluster
- Upgrade to 18.2.x (From 17.11.4 to 18.2.0 and 18.0.4 to 18.2.1)
What is the current bug behavior?
- Gitaly clusters are in a failed state (Environment 1)
- Browsing to GitLab pages results in HTTP 502 i.e unavailable (Environment 2)
What is the expected correct behavior?
- Gitaly Clusters to be in a healthy / working state (Environment 1)
- GitLab pages is available upon browsing (Environment 2)
Relevant logs and/or screenshots
Environment 1
Upgrade from: 17.11.4 to 18.2.0
On this environment (air-gapped), we see some handshake errors in the NGINX logs:
*21 Upstream server temporarily disabled while SSL handshaking to upstream, client <gitlabIP>, server, 0.0.0.0:3305, upstream <praefectip>:3305
*21 connect() failed (111: connection refused) while connecting upstream, client: <gitlabIP>, server 0.0.0.0:3305, upstream:"<praefectip>:3305"Looking in the Praefect logs, we find these entries:
{"level":"warning","
msg":"[core][channel #1 SubChannel #4] grpc: addrConn.createTransport failed to connect to {Addr: \"<gitalyip>:9999\",
ServerName:\"<gitalyhostname>:9999\",
BalancerAttributes: {\"\u003c%!p(pickfirstleaf.managedByPickfirstKeyType={})\u003e\": \"\u003c%!p(bool=true)\u003e\"}} Err: connection error: desc = \"transport: authentication handshake failed crypto/edch: use of x25519 is not allowed in FIPS 140-only mode \""
,"pid":<pid>,
"system":"system",
"time","<time>"}
{"component","HealthManager",
"correlation_id":"<id>",
"error":"rpc error: code = Unavailable desc = connection error: desc = \" transport: authentication handshake failed: crypto/edch: use of x25519 is not allowed in FIPS 140-only mode\"",
"level":"error",
"msg":"failed checking node health:,
"pid":"<pid>",
"storage":"<gitaly_storage_name>",
"time":"<time>",
"virtual_storage":"default"
}The message that sticks out is transport: authentication handshake failed: crypto/edch: use of x25519 is not allowed in FIPS 140-only mode when Praefect and Gitaly clusters are already FIPS enabled.
Environment 1 also has the following NGINX conf file for their internal load balancer:
Stream {
upstream praefecttls {
server <praefect_1>.<domain>:3305
server <praefect_2>.<domain>:3305
server <praefect_3>.<domain>:3305
}
server {
listen 3305 ssl;
proxy_pass praefecttls;
ssl_certificate /etc/ssl/certs/<nginxcert>.pem;
ssl_certificate_key /etc/ssl/certs/<nginxkey>.key;
ssl_alpn http/1.1, h2;
proxy_ssl_server_name on;
proxy_ssl on;
proxy_ssl_certificate /etc/ssl/certs/<praefectcert>.pem;
ssl_password_file <passwordfilepath>;
proxy_ssl_certificate_key /etc/ssl/certs/<praefectkey>.key;
proxy_ssl_protocols TLSv1.3;
ssl_protocols TLSv1.3;
ssl_prefer_server_ciphers off;
proxy_ssl_session_reuse on;
}
}Additionally, protocols on the NGINX load balancer were set to only use FIPS compliant ciphers in the gitlab.rb file per the nginx['ssl_ciphers'] option thats referenced in the documentation
Environment 2
Upgraded from: 18.0.4 to 18.2.1
On the second environment (not air-gapped), we see HTTP 502 statuses in the /var/log/gitlab/gitlab-pages/current logs along with an error message saying that a used cipher is not FIPS compliant:
{
"correlation_id":"<CORRELATION_ID>",
"error":"Get \"<GITLAB_URL>": crypto/ecdh: use of X25519 is not allowed in FIPS 140-only mode",
"host":"<GITLAB_HOST>",
"level":"error",
"msg":"could not fetch domain information from a source",
"path":"/docs","time":"2025-07-30T08:11:07-04:00"}
{
"content_type":"text/html; charset=utf-8",
"correlation_id":"<CORRELATION_ID>",
"duration_ms":3135,
"host":"<GITLAB_HOST>",
"level":"info",
"method":"GET",
"msg":"access",
"pages_https":true,
"proto":"HTTP/1.1",
"read_bytes":995,
"referrer":"<REFERRER>",
"remote_addr":"<IP>",
"remote_ip":"",
"status":502,
"system":"http",
"time":"2025-07-30T08:11:07-04:00",
"ttfb_ms":3135,
"uri":"/docs",
"user_agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/138.0.0.0 Safari/537.36 Edg/138.0.0.0",
"written_bytes":3563
}
{
"correlation_id":"<CORRLEATION_ID>",
"host":"<HOST>",
"level":"info",
"middleware_time_distribution":{"Acme_ms":0,
"CustomHeaders_ms":0,
"HTTPSRedirect_ms":0,
"HealthCheck_ms":0,
"Metrics_ms":0,
"RateLimiter_ms":0,
"Routing_ms":3135,
"cors_ms":0},
"msg":"Request time exceeded 0.5 seconds",
"other_time_distribution":{},
"path":"/docs",
"time":"2025-07-30T08:11:07-04:00",
"total_execution_time_ms":3135
}
{
"correlation_id":"<CORRELATION_ID>"
,"error":"Get \"<URL>": crypto/ecdh: use of X25519 is not allowed in FIPS 140-only mode",
"host":"<HOST>",
"level":"error",
"msg":"could not fetch domain information from a source",
"path":"",
"time":"2025-07-30T08:11:10-04:00"
}
{
"content_type":"text/html; charset=utf-8",
"correlation_id":"<CORRELATION_ID>",
"duration_ms":3031,
"host":"<HOST>",
"level":"info",
"method":"GET",
"msg":"access",
"pages_https":true,
"proto":"HTTP/1.1",
"read_bytes":851,
"referrer":"<REFERRER>",
"remote_addr":"<IP>",
"remote_ip":"",
"status":502,
"system":"http",
"time":"2025-07-30T08:11:10-04:00",
"ttfb_ms":3031,
"uri":"",
"user_agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/138.0.0.0 Safari/537.36 Edg/138.0.0.0","written_bytes":3563}
{
"correlation_id":"<CORRELATION_ID>",
"host":"<HOST>",
"level":"info",
"middleware_time_distribution":
{
"Acme_ms":0,
"CustomHeaders_ms":0,
"HTTPSRedirect_ms":0,
"HealthCheck_ms":0,
"Metrics_ms":0,
"RateLimiter_ms":0,
"Routing_ms":3031,
"cors_ms":0},
"msg":"Request time exceeded 0.5 seconds",
"other_time_distribution":{},
"path":"",
"time":"2025-07-30T08:11:10-04:00",
"total_execution_time_ms":3031
}The message that sticks out here is Get \"<GITLAB_URL>": crypto/ecdh: use of X25519 is not allowed in FIPS 140-only mode
Output of checks
Results of GitLab environment info
Expand for output related to GitLab environment info
(For installations with omnibus-gitlab package run and paste the output of: `sudo gitlab-rake gitlab:env:info`) (For installations from source run and paste the output of: `sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production`)
Results of GitLab application Check
Expand for output related to the GitLab application check
(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:check SANITIZE=true)(For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true)(we will only investigate if the tests are passing)
Possible fixes
Patch release information for backports
If the bug fix needs to be backported in a patch release to a version under the maintenance policy, please follow the steps on the patch release runbook for GitLab engineers.
Refer to the internal "Release Information" dashboard for information about the next patch release, including the targeted versions, expected release date, and current status.
High-severity bug remediation
To remediate high-severity issues requiring an internal release for single-tenant SaaS instances, refer to the internal release process for engineers.