Terraform template: caching `.terraform.lock.hcl` is causing issues

Summary

Using the out-of-the box GitLab Terraform CI/CD template causes issues because it caches .terraform.lock.hcl.

Steps to reproduce

• This was already discussed in this thread on the merge request: !50647 (comment 581575028)
• There is a support ticket for this: https://support.gitlab.com/hc/en-us/requests/224627

Update a module

1. Use Terraform locally and in CI/CD
2. Commit the .terraform.lock.hcl
3. Run pipeline
4. Update version requirements, commit updated .terraform.lock.hcl
5. Running the pipeline, the .terraform.lock.hcl from the repo will be overwritten by the cache
6. Pipeline fails

Use different runners

1. Use multiple runners (GitLab shared + private ones)
2. init job is scheduled on another family of runners than the rest of the pipeline
3. Because of the not-shared cache, cached files are missing and pipeline fails

Example Project

Internal project can be provided on request.

This was also investigated by GitLab support. https://support.gitlab.com/hc/en-us/requests/224627

What is the current bug behavior?

• Pipeline fails because of .terraform.lock.hcl caching

What is the expected correct behavior?

• Pipeline should not fail
• I don't think it's a good practice to cache .terraform.lock.hcl if it is checked-in to the git repository
• In my opinion, that's a mis-use of caching altogether. In my understanding, a cache should speed up things, if the cached value is not present, it should be possible to recover from it by just calculating the missing value. As the Terraform template is designed right now, this is not the case, downstream CI jobs in the pipeline rely on the cached items being present.

Relevant logs and/or screenshots

Output of checks

This bug happens on GitLab.com