fix(datastore): race condition pushing two or more manifests by tag with the same digest

Summary

I am getting an error when trying to build a docker image and publish it to container registry

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':jib'.
> com.google.cloud.tools.jib.plugins.common.BuildStepsExecutionException: 500 Internal Server Error
  {"errors":[{"code":"UNKNOWN","message":"unknown error","detail":{}}]}

We have more than 50 projects in the repository and most of them are working fine. They are all built the same way. This issue affects new projects.

Steps to reproduce

1. create new project in gitlab (ex. test-project-1)
2. get working project (which has successful pipeline) and add origin from test-project-1
3. push project
4. pipeline fails

Example Project

What is the current bug behavior?

Pipeline failed. I noticed some interesting logs after enabling debug mode in jib. As I mentioned before - there is a problem only with new projects (with the same dependencies like jib, spring, java version etc.).

Relevant logs and/or screenshots

2022-03-18T16:58:44.764+0000 [INFO] [com.google.api.client.http.HttpTransport] -------------- REQUEST  --------------
PUT https://registry.gitlab.com/v2/<group>/<project>/manifests/clean-project
Accept: 
Accept-Encoding: gzip
Authorization: <Not Logged>
User-Agent: jib 3.2.0 jib-gradle-plugin Google-HTTP-Java-Client/1.34.0 (gzip)
Content-Type: application/vnd.docker.distribution.manifest.v2+json

2022-03-18T16:58:44.764+0000 [INFO] [com.google.api.client.http.HttpTransport] curl -v --compressed -X PUT -H 'Accept: ' -H 'Accept-Encoding: gzip' -H 'Authorization: <Not Logged>' -H 'User-Agent: jib 3.2.0 jib-gradle-plugin Google-HTTP-Java-Client/1.34.0 (gzip)' -H 'Content-Type: application/vnd.docker.distribution.manifest.v2+json' -d '@-' -- 'https://registry.gitlab.com/v2/<group>/<project>/manifests/clean-project' << $$$
2022-03-18T16:58:44.797+0000 [DEBUG] [jdk.event.security] ValidationChain: -347365895, -1835097136, -1473757959
2022-03-18T16:58:44.799+0000 [DEBUG] [jdk.event.security] ValidationChain: -347365895, -1835097136, -1473757959
2022-03-18T16:58:44.802+0000 [DEBUG] [jdk.event.security]  TLSHandshake: registry.gitlab.com:443, TLSv1.3, TLS_AES_256_GCM_SHA384, -1473757959
2022-03-18T16:58:44.803+0000 [DEBUG] [jdk.event.security]  TLSHandshake: registry.gitlab.com:443, TLSv1.3, TLS_AES_256_GCM_SHA384, -1473757959
2022-03-18T16:58:44.806+0000 [INFO] [com.google.api.client.http.HttpTransport] Total: 1,403 bytes
2022-03-18T16:58:44.806+0000 [INFO] [com.google.api.client.http.HttpTransport] {"schemaVersion":2,"mediaType":"application/vnd.docker.distribution.manifest.v2+json","config":{"mediaType":"application/vnd.docker.container.image.v1+json","digest":"sha256:c62a83f5a8b0cc102cefda11e3343e2709de5b4070e7612f42c3bdaeee8e35e1","size":6008},"layers":[{"mediaType":"application/vnd.docker.image.rootfs.diff.tar.gzip","digest":"sha256:36ccefbf3d8a9a1b18baaa9cbf0f3ad50e8a7b751656c74df359900a318cbffc","size":2816169},{"mediaType":"application/vnd.docker.image.rootfs.diff.tar.gzip","digest":"sha256:98cdb5547088b654845fc6a44678898e6d0af07d4301b4ceb54c62175b128947","size":6486650},{"mediaType":"application/vnd.docker.image.rootfs.diff.tar.gzip","digest":"sha256:f9543a44faf48a8db5867137bb2ecf9d154016c8281823a97526d8493aa603c7","size":5094},{"mediaType":"application/vnd.docker.image.rootfs.diff.tar.gzip","digest":"sha256:b73661c4b41c93232bb8f747bf51317c3a0d1abbc36c7bdf8496d210656e7864","size":138626713},{"mediaType":"application/vnd.docker.image.rootfs.diff.tar.gzip","digest":"sha256:bf5bec95ba76db36fafe8e3ebac630dc4cf7ab763a5442d104bc8a8756ab7360","size":7581779},{"mediaType":"application/vnd.docker.image.rootfs.diff.tar.gzip","digest":"sha256:e0240c0a3ceee5ce8fc9fad9518dd1b4794d0bdb79da2bd9dd8d741c625566e9","size":945},{"mediaType":"application/vnd.docker.image.rootfs.diff.tar.gzip","digest":"sha256:7e75240879d4c8b9816b89b59fa09cdbb0a59e2e0189a95a008d508fc20a6fda","size":401}]}
2022-03-18T16:58:44.807+0000 [INFO] [com.google.api.client.http.HttpTransport] Total: 1,403 bytes
2022-03-18T16:58:44.807+0000 [INFO] [com.google.api.client.http.HttpTransport] {"schemaVersion":2,"mediaType":"application/vnd.docker.distribution.manifest.v2+json","config":{"mediaType":"application/vnd.docker.container.image.v1+json","digest":"sha256:c62a83f5a8b0cc102cefda11e3343e2709de5b4070e7612f42c3bdaeee8e35e1","size":6008},"layers":[{"mediaType":"application/vnd.docker.image.rootfs.diff.tar.gzip","digest":"sha256:36ccefbf3d8a9a1b18baaa9cbf0f3ad50e8a7b751656c74df359900a318cbffc","size":2816169},{"mediaType":"application/vnd.docker.image.rootfs.diff.tar.gzip","digest":"sha256:98cdb5547088b654845fc6a44678898e6d0af07d4301b4ceb54c62175b128947","size":6486650},{"mediaType":"application/vnd.docker.image.rootfs.diff.tar.gzip","digest":"sha256:f9543a44faf48a8db5867137bb2ecf9d154016c8281823a97526d8493aa603c7","size":5094},{"mediaType":"application/vnd.docker.image.rootfs.diff.tar.gzip","digest":"sha256:b73661c4b41c93232bb8f747bf51317c3a0d1abbc36c7bdf8496d210656e7864","size":138626713},{"mediaType":"application/vnd.docker.image.rootfs.diff.tar.gzip","digest":"sha256:bf5bec95ba76db36fafe8e3ebac630dc4cf7ab763a5442d104bc8a8756ab7360","size":7581779},{"mediaType":"application/vnd.docker.image.rootfs.diff.tar.gzip","digest":"sha256:e0240c0a3ceee5ce8fc9fad9518dd1b4794d0bdb79da2bd9dd8d741c625566e9","size":945},{"mediaType":"application/vnd.docker.image.rootfs.diff.tar.gzip","digest":"sha256:7e75240879d4c8b9816b89b59fa09cdbb0a59e2e0189a95a008d508fc20a6fda","size":401}]}
2022-03-18T16:58:44.856+0000 [INFO] [com.google.api.client.http.HttpTransport] -------------- RESPONSE --------------
HTTP/1.1 500 Internal Server Error
Content-Type: application/json
Docker-Distribution-Api-Version: registry/2.0
Gitlab-Migration-Path: new
X-Content-Type-Options: nosniff
Date: Fri, 18 Mar 2022 16:58:44 GMT
Content-Length: 70

2022-03-18T16:58:44.856+0000 [INFO] [com.google.api.client.http.HttpTransport] Total: 70 bytes
2022-03-18T16:58:44.856+0000 [INFO] [com.google.api.client.http.HttpTransport] {"errors":[{"code":"UNKNOWN","message":"unknown error","detail":{}}]}

Status

Current as of 2022-04-08 #640 (comment 905432125)