502 when listing issues of big repositories
Summary
We're trying to retrieve all pages of the issue listing API endpoint for https://gitlab.com/gitlab-org/gitlab-ce/ using IGitt: https://pypi.python.org/pypi/IGitt/0.4.1.dev20171011100142
This is hindering us at GitMate.io, CC @teemo and @victorwu
When doing that we get an error 502 multiple times. Implementing exponential backoff in IGitt hasn't helped.
Steps to reproduce
In [1]: from IGitt.GitLab.GitLabRepository import GitLabRepository, GitLabPrivateToken
In [2]: token = GitLabPrivateToken('<private token here>')
In [3]: repo = GitLabRepository(token, 'gitlab-org/gitlab-ee')
In [4]: repo.filter_issues(state='all')
---------------------------------------------------------------------------
RuntimeError Traceback (most recent call last)
<ipython-input-4-ba99a2a52afc> in <module>()
----> 1 repo.filter_issues(state='all')
/home/lasse/.venvs/coon/lib/python3.6/site-packages/IGitt/GitLab/GitLabRepository.py in filter_issues(self, state)
383 return {GitLabIssue.from_data(res, self._token,
384 self.full_name, res['iid'])
--> 385 for res in get(self._token, self._url + '/issues', params)}
386
387 @property
/home/lasse/.venvs/coon/lib/python3.6/site-packages/IGitt/GitLab/__init__.py in get(token, url, params, headers)
116 return _fetch(BASE_URL, 'get', token,
117 url, query_params={**dict(params or {}), 'per_page': 100},
--> 118 headers=headers)
119
120
/home/lasse/.venvs/coon/lib/python3.6/site-packages/IGitt/Interfaces/__init__.py in _fetch(base_url, req_type, token, url, data, query_params, headers)
102 }
103 method = req_methods[req_type]
--> 104 resp = get_response(method, base_url + url, json=data)
105
106 # DELETE request returns no response
/home/lasse/.venvs/coon/lib/python3.6/site-packages/backoff/_sync.py in retry(*args, **kwargs)
83
84 try:
---> 85 ret = target(*args, **kwargs)
86 except exception as e:
87 if giveup(e) or tries == max_tries_:
/home/lasse/.venvs/coon/lib/python3.6/site-packages/IGitt/Interfaces/__init__.py in get_response(method, *args, **kwargs)
69 response = method(*args, **kwargs)
70 if response.status_code >= 300:
---> 71 raise RuntimeError(response, response.status_code)
72 return response
73
RuntimeError: (<Response [502]>, 502)pip freeze | grep IGitt
IGitt==0.4.1.dev20171011100142What is the current bug behavior?
Error 502
What is the expected correct behavior?
Get an issue list
Relevant logs and/or screenshots
See reproduction
Output of checks
This bug happens on GitLab.com
Results of GitLab environment info
GitLab.com
Results of GitLab application Check
?
Possible fixes
nope