Cache golang modules between builds

For the past day, we've been seeing an elevated rate of failures of the setup-test-env job, with various errors from go get when trying to download modules needed to compile the gitlab-elasticsearch-indexer or Gitaly projects. For example, this pipeline: https://gitlab.com/gitlab-org/gitlab/pipelines/83310644

We're also seeing similar failures on other jobs around GitLab, suggesting a general problem, e.g. in CNG: https://gitlab.com/gitlab-org/build/CNG/-/jobs/300172610

For the gitlab project, specifically, we can (and should) be caching the files downloaded by go get into $GOPATH/pkg/mod as part of the build, per our Go guidelines: https://docs.gitlab.com/ee/development/go_guide/index.html#modules

Our pipeline is, however, pretty complicated 😬 - some documentation on it can be found here: https://docs.gitlab.com/ee/development/pipelines.html

I've got the start of an MR in !17350 (diffs) but need to end for the day, so I'm creating this issue for visibility.

Failure modes vary, but include the following complaints from go get:

• connection refused messages
• io timeout messages
• unknown revision messages

The third, in particular, is not very obvious - I assume a TCP connection error after verifying the dependency exists in-general, but while trying to discover if a specific tag exists, is to blame.

Another thing we could try is bumping the go version we use to go1.13 from go1.11 - the go get code has been significantly reworked in the meantime, and there may be some improvements.

We can also consider setting the GOPROXY environment variable, perhaps to: https://goproxy.io/ ?