Skip to content

Client_max_body_size not updating every vhost

Summary

I'm trying to increase the max size of object allowed. I've set proxy-body-size in everywhere I can find documentation for. The resulting nginx.conf on the gitlab-nginx-ingress-controller pod only has the client_max_body_size is only changed on the default vhost/server entry. If I manually update the other instances of client_max_body_size in nginx.conf on the pod, and kill -HUP the nginx process I can push larger objects.

Configuration used

host:
  domain: mydomain.com
nginx-ingress:                                                                                                                                                                                                                                                 
  enabled: true                                                                                                                                                                                                                                                
  tcpExternalConfig: "true"                                                                                                                                                                                                                                    
  controller:                                                                                                                                                                                                                                                  
    addHeaders:                                                                                                                                                                                                                                                
      Referrer-Policy: strict-origin-when-cross-origin                                                                                                                                                                                                         
    config:                                                                                                                                                                                                                                                    
      hsts: "false"                                                                                                                                                                                                                                            
      hsts-include-subdomains: "false"                                                                                                                                                                                                                         
      server-name-hash-bucket-size: "256"                                                                                                                                                                                                                      
      use-http2: "true"                                                                                                                                                                                                                                        
      ssl-ciphers: "ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-AES256-SHA:ECDHE-RSA-AES128-SHA:AES256-GCM-SHA384:AES128-GCM-SHA256:AES256-SHA256:AES128-SHA256:AES256-SHA:AES128-SHA:!aNULL:!eNULL:!EXPORT:!DES:!MD5:!PSK:!RC4"                                                                                                                                                            
      ssl-protocols: "TLSv1.3 TLSv1.2"                                                                                                                                                                                                                         
      server-tokens: "false"                                                                                                                                                                                                                                   
      proxy-real-ip-cidr: 10.128.0.0/16                                                                                                                                                                                                                        
      use-proxy-protocol: "true"                                                                                                                                                                                                                               
      proxy-body-size: "4096m"                                                                                                                                                                                                                                 
    service:                                                                                                                                                                                                                                                   
      externalTrafficPolicy: "Local"                                                                                                                                                                                                         
      annotations:                                                                                                                                                                                                                                             
        kubernetes.io/ingress.class: "internal-ingress"                                                                                                                                                                                                        
        external-dns.alpha.kubernetes.io/hostname: "gitlab.mydomain.com"                                                                                                                                                                                       
        external-dns.alpha.kubernetes.io/target: "gitlab.aws.wevr.com"                                                                                                                                                                                         
        service.beta.kubernetes.io/aws-load-balancer-internal: "true"                                                                                                                                                                                          
        service.beta.kubernetes.io/aws-load-balancer-type: "nlb-ip"                                                                                                                                                                                            
        service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"                                                                                                                                                                 
        service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp                                                                                                                                                                                     
        service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: "*"                                                                                                                                                                                       
        service.beta.kubernetes.io/aws-load-balancer-ssl-cert: "XXXXXX"                                                                                                           
        service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "443"                                                                                                                                                                                          
        nginx.ingress.kubernetes.io/client-body-buffer-size: "2M"                                                                                                                                                                                              
        nginx.ingress.kubernetes.io/proxy-body-size: "4096m"                                                                                                                                                                                                   
        igress.kubernetes.io/proxy-body-size: "4096m"     
global:
    ingress:                                                                                                                                                                                                                                                     
    configureCertmanager: false                                                                                                                                                                                                                                
    service:                                                                                                                                                                                                                                                   
      annotations:                                                                                                                                                                                                                                             
        nginx.ingress.kubernetes.io/client-body-buffer-size: "2M"                                                                                                                                                                                              
        nginx.ingress.kubernetes.io/proxy-body-size: "4096m"                                                                                                                                                                                                
        igress.kubernetes.io/proxy-body-size: "4096m"                                                                                                                                                                                                       
      proxyBodySize: "4096m"                                                                                                                                                                                                                                
    enabled: true                                                                                                                                                                                                                                              
    tls:                                                                                                                                                                                                                                                       
      enabled: false                                                                                                                                                                                                                                           
    path: /                                 

Current behavior

client_max_body_size gets updated in the the default server block of the nginx config, but not the vhost block. I can only push objects up to 512mb to my gitlab instance. I get an http 413 error.

...
server {
  server_name _;
  ## This gets updated
  location / {
    client_max_body_size: 4096m;
   }
}
...
server {
 server_name gitlab.mydomain.com;
 ## This does not get updated
 location / {
   client_max_body_size: 512m;
 }
}
...

Expected behavior

Have each relevant client_max_body_size in the config reflect the specified value.

Versions

  • Chart:
helm list | grep gitlab
gitlab          default         19              2021-04-13 18:53:26.341070697 -0700 PDT deployed        gitlab-4.10.2         13.10.2    
  • Platform:
    • Cloud: EKS
  • Kubernetes: (kubectl version)
    • Client: 1.18
    • Server: 1.19
  • Helm: (helm version)
    • Client: 3.5.3
    • Server:

Relevant logs

Log From the gitlab-nginx-ingress-controller after exceeding the 512mb limit to an object.

565080a39f654abc718ce5c2e7fdc34/1048576000 HTTP/1.1" 413 176 "-" "git-lfs/2.8.0 (Fedora 30; linux amd64; go 1.12.7)" 537006742 40.718 [default-gitlab-webservice-default-8181] [] - - - - 0bb5d427e4afdf858e2ce80af418d3cc