New geo deployments fail to initialise tracking database (cloud native)
Note that this issue is a call for help more than anything else. The issue described has not been reliably reproduced.
Reports started coming in recently among the users of GET that the new geo deployments raise the following error:
kubectl -n default logs gitlab-geo-logcursor-$podsuffixes -c dependencies
...
Database Schema - geo (gitlabhq_geo_production) - current: 0, codebase: 20230808031211
NOTICE: Database has not been initialized yet.
Important thing to note is that the issue is intermittent.
The workaround currently in use is to run:
kubectl -n default exec -it $(kubectl -n default get pods -l app=toolbox -o name) -- env DISABLE_DATABASE_ENVIRONMENT_CHECK=1 gitlab-rake db:schema:load:geo
It is understood that this workaround is not suitable for production use.
Is there anything that changed product-wise that might be contributing to this new behaviour we are observing? Are there any tips GEO team can share with us?