Symbols are not rendered correctly when using Reverse Proxies and Relative URL

I'm trying to use the new Relative URL support behind a reverse proxy. It seems to work on a basic level, but Gitlab fails to load the font symbols:

This is my gitlab.rb:

external_url 'https://servername.com/gitlab'
[...]
################
# GitLab Nginx #
################
[...]
nginx['listen_port'] = 442
nginx['listen_https'] = false 

My nginx configuration:

# Gitlab
  location /gitlab {
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_pass http://localhost:442;
    rewrite ^/gitlab$ /gitlab/ permanent;
  }

As soon as I disable the reverse Proxy and reconfigure Gitlab's nginx instance to use HTTPS but still using a relative URL, the symbols are rendered correctly.