Improve pipeline for status page project to avoid so many concurrent requests to raw endpoint
This script - https://gitlab.com/gitlab-org/status-page/-/blob/master/scripts/sync_emojis.js - is used by a pipeline that runs every so often and has caused performance issues in the past for Gitaly cny: #661 (comment 450955781)
The basic problem is that it performs over 1,000 concurrent HTTP requests to the raw endpoint. It does this to get the contents of https://gitlab.com/gitlab-org/gitlab/-/tree/master/app/assets/images/emoji.
I think we could improve this pipeline in two ways:
- Use a more specific ref than
master
. This pipeline would probably not work if we delete that directory from the GitLab Rails source in future. - Use the archive endpoint to fetch the entire directory at once, like https://gitlab.com/gitlab-org/gitlab/-/archive/master/gitlab-master.zip?path=app/assets/images/emoji
We should only do this after #677 (closed), because this pipeline - while frustrating - is a good load-test