GitLab.com: Repositories root in GCS bucket treated as a potential repository during enumeration
Context
I found this while working on gitlab-com/gl-infra/delivery#1729 (comment 581837904).
Problem
It seems that the root directory (docker/registry/v2/repositories/
) of the prod bucket is being treated as a repository "folder" rather than a simple path prefix, which is causing an index out of bounds error here, because filePath
and root
are the same (docker/registry/v2/repositories/
):
panic: runtime error: slice bounds out of range [33:32]
goroutine 1 [running]:
github.com/docker/distribution/registry/storage.handleRepository(0x177ce40, 0xc0001e8000, 0xc0002600a0, 0x20, 0x0, 0x0, 0xc000212980, 0x0, 0x0)
/home/jpereira/2021-05-21/src/registry/storage/catalog.go:143 +0x3e8
github.com/docker/distribution/registry/storage.(*registry).Enumerate.func1(0x177ce40, 0xc0001e8000, 0x1fa50a0, 0xc0003c4780)
/home/jpereira/2021-05-21/src/registry/storage/catalog.go:68 +0x58
github.com/docker/distribution/registry/storage/driver.WalkFallback(0x177bfd0, 0x1fa50a0, 0x178d8f8, 0xc0001de0e0, 0xc0002600a0, 0x20, 0xc000210120, 0xc00057fa38, 0x40f8bb)
/home/jpereira/2021-05-21/src/registry/storage/driver/walk.go:48 +0x2d6
github.com/docker/distribution/registry/storage/driver/gcs.(*driver).Walk(...)
/home/jpereira/2021-05-21/src/registry/storage/driver/gcs/gcs.go:965
github.com/docker/distribution/registry/storage/driver/gcs.(*driver).WalkParallel(0xc0001de0e0, 0x177bfd0, 0x1fa50a0, 0xc0002600a0, 0x20, 0xc000210120, 0x7efcbeed4a68, 0x20)
/home/jpereira/2021-05-21/src/registry/storage/driver/gcs/gcs.go:973 +0xf6
github.com/docker/distribution/registry/storage.(*registry).Enumerate(0xc000476090, 0x177bfd0, 0x1fa50a0, 0xc000212980, 0x0, 0xc0001de230)
/home/jpereira/2021-05-21/src/registry/storage/catalog.go:67 +0xe6
github.com/docker/distribution/registry/storage/inventory.(*Taker).Run(0xc0005b40c0, 0x177bfd0, 0x1fa50a0, 0xc000070c50, 0x0, 0x0)
/home/jpereira/2021-05-21/src/registry/storage/inventory/inventory.go:107 +0x255
github.com/docker/distribution/registry.glob..func11(0x1f92b20, 0xc000384b10, 0x1, 0x3)
/home/jpereira/2021-05-21/src/registry/root.go:591 +0x43d
github.com/spf13/cobra.(*Command).execute(0x1f92b20, 0xc000384ae0, 0x3, 0x3, 0x1f92b20, 0xc000384ae0)
/home/jpereira/go/pkg/mod/github.com/spf13/cobra@v1.1.3/command.go:856 +0x2c2
github.com/spf13/cobra.(*Command).ExecuteC(0x1f928a0, 0x0, 0xffffffff, 0xc000090058)
/home/jpereira/go/pkg/mod/github.com/spf13/cobra@v1.1.3/command.go:960 +0x375
github.com/spf13/cobra.(*Command).Execute(...)
/home/jpereira/go/pkg/mod/github.com/spf13/cobra@v1.1.3/command.go:897
main.main()
/home/jpereira/2021-05-21/src/cmd/registry/main.go:21 +0x2d
It's the first time we're seeing an error like this. I know that we moved the prod registry bucket from S3 to GCS some time ago. I don't know which tool we used to do so, but I'm afraid we might be running into something like #341 (closed). In this case, the tool used to move the registry data has written zero byte metadata files during the transfer, and that tricks the registry.
Note: Repository enumeration is not something that the registry does outside of offline GC and the /v2/_catalog
route. We use none of these for GitLab.com, thus this problem has no impact on the day-to-day operation.
Solution
The quickest way to unblock gitlab-com/gl-infra/delivery#1729 (closed) is to skip the evaluation in case filePath
is the repositories root. That will also show us if the problem is only with the root "directory" or if there are other "directories" affected (this is currently the first one evaluated, so the inventory execution stops right there).