create a dedicated buildbot
instead of gitlab's shared runners. probably ec2 or gce.
if this doesn't help, consider moving back to circleci and bitbucket. I like github pipelines and the flexibility of gitlab CI (seriously, it's fantastic that dedicated runners are even possible with hosted CI), but circleCI was faster out of the box and cached correctly.