Timeouts (30s) occurring with long requests on chart install?
Summary
Hey. Recently deployed a default Helm install of GitLab on GKE and it's generally been working well. However, when trying to test out the project import API I noticed that either a timeout or socket closure was occurring after 30s (specifically Broken Pipe error).
Been trying to debug this and I found I also see the same behaviour via the UI but in that case the browser reports a err_connection_reset.
Steps to reproduce
Upload a large project export via API or UI. Notice that it fails after 30 seconds.
I used the following python script (adapted from the Project Import docs) to upload this gitlabhq project export file
import-project.py
import requests
import urllib
import json
import sys
url = 'http://<URL>/api/v4/projects/import'
data = {
"path": "gitlabhq",
"namespace": "qa-perf-testing"
}
headers = {
'Private-Token': "<API_TOKEN>"
}
with open('<PROJECT_TARBALL_FILEPATH>', 'rb') as f:
requests.post(url, headers=headers, data=data, files={'<PROJECT_TARBALL_FILENAME>': f})
import-project.py
outputtime python3 import-project-py.py
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 603, in urlopen
chunked=chunked)
File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 355, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1229, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1275, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1224, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1055, in _send_output
self.send(chunk)
File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 977, in send
self.sock.sendall(data)
BrokenPipeError: [Errno 32] Broken pipe
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/requests/adapters.py", line 449, in send
timeout=timeout
File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 641, in urlopen
_stacktrace=sys.exc_info()[2])
File "/usr/local/lib/python3.7/site-packages/urllib3/util/retry.py", line 368, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/usr/local/lib/python3.7/site-packages/urllib3/packages/six.py", line 685, in reraise
raise value.with_traceback(tb)
File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 603, in urlopen
chunked=chunked)
File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 355, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1229, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1275, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1224, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1055, in _send_output
self.send(chunk)
File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 977, in send
self.sock.sendall(data)
urllib3.exceptions.ProtocolError: ('Connection aborted.', BrokenPipeError(32, 'Broken pipe'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "import-project-py.py", line 16, in <module>
requests.post(url, headers=headers, data=data, files={'2019-07-18_15-58-640_qa-perf-testing_gitlabhq_export.tar.gz': f})
File "/usr/local/lib/python3.7/site-packages/requests/api.py", line 116, in post
return request('post', url, data=data, json=json, **kwargs)
File "/usr/local/lib/python3.7/site-packages/requests/api.py", line 60, in request
return session.request(method=method, url=url, **kwargs)
File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 533, in request
resp = self.send(prep, **send_kwargs)
File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 646, in send
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python3.7/site-packages/requests/adapters.py", line 498, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', BrokenPipeError(32, 'Broken pipe'))
**python3 import-project-py.py 0.33s user 0.91s system 4% cpu 30.227 total**
Note the total time of 30.277
seconds which suggests there's a 30s timeout somewhere on the server.
Configuration used
Standard Helm deploy. Only IP and Hostname change.
Current behavior
Attempting to make a long call over 30s, e.g. uploading a larger file (600m), times out at 30s.
Expected behavior
That I can upload larger files (600mb or more) like I can on other GitLab environments with a default Helm chart deploy on GKE.
Versions
- Chart: 2.0.0
- Platform:
- Cloud: GKE
- Kubernetes: (
kubectl version
)- Client:
v1.12.9-gke.7
- Server:
v1.12.8-gke.10
- Client:
- Helm: (
helm version
)- Client:
&version.Version{SemVer:"v2.14.2", GitCommit:"a8b13cc5ab6a7dbef0a58f5061bcc7c0c61598e7", GitTreeState:"clean"}
- Server:
&version.Version{SemVer:"v2.14.2", GitCommit:"a8b13cc5ab6a7dbef0a58f5061bcc7c0c61598e7", GitTreeState:"clean"}
- Client: