Too many labels in one rule causes 502
We have a situation where a rule for closing stale tickets has plenty (35 to be exact) labels in the forbidden_labels
section (multiple teams using the same project issue tracker for their kanban boards). When we added the last one or two labels we started getting 502 from our self hosted Gitlab instance. At first I thought the request timed out but it didn't make sense because it happened almost immediately. After setting the --debug
flag in the command it turned out that it did get some issues processed but got 502 on the second page:
.[DEBUG] query_api: https://gitlab.example.com/api/v4/projects/example%2Fexample/issues?per_page=100&labels=Vintage¬%5Blabels%5D=<plenty of labels>&state=opened&updated_before=2022-05-16
[DEBUG] rate_limit_infos: Rate limit remaining: 0 (reset at 1970-01-01 02:00:00 +0200)
[DEBUG] Rate limit almost exceeded, sleeping for -1653930241.565096 seconds
.[DEBUG] query_api: https://gitlab.example.com/api/v4/projects/example%2Fexample/issues?per_page=100&labels=Vintage¬%5Blabels%5D=<plenty of labels>&state=opened&updated_before=2022-05-16&page=2
/Library/Ruby/Gems/2.6.0/gems/gitlab-triage-1.23.1/lib/gitlab/triage/network.rb:107:in `raise_unexpected_response': <HTML for the 502 page>
Took a look at the gitlab_error.log
and the 502 turned out to be caused by too long header:
==> /var/log/gitlab/nginx/gitlab_error.log <==
2022/05/30 16:57:06 [error] 16599#0: *1220 upstream sent too big header while reading response header from upstream
With some google-fu I ran across with a similar but unrelated issue in the Gitlab issue tracker and a fix for it:
gitlab-org/gitlab#351181 (closed)
gitlab-org/omnibus-gitlab#6678 (closed)
So the fix in our case was just adding the following to the /etc/gitlab/gitlab.rb
:
nginx['proxy_custom_buffer_size'] = '16k'
Don't know if this really requires any other fix than that, maybe add a mention about it in the README.md? Anyway, thought to open a ticket to save somebody else from some head scratching.