See https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/1976#note_30750187
/cc @stanhu @brodock @rspeicher