git2go.UpdateFile commit: gob: name not registered for interface
Background Information
During gitlab-com/gl-infra/production#4810 (closed) it was noted that the error above was elevated during the deployment time.
It was also seen in a drifted node file-10
up until it's version was put back in sync with the rest of the fleet.
The full error message as seen in Kibana logs:
commit: commit: gob: name not registered for interface: "gitlab.com/gitlab-org/gitaly/internal/git2go.UpdateFile"
Sentry bug report: https://sentry.gitlab.net/gitlab/gitaly-production/issues/158811/?query=is%3Aunresolved%20git2go.UpdateFile
Related Kibana logs: https://log.gprd.gitlab.net/goto/369fa0d43288f94d35039ca3a256007b
UI error as reported by end user (@furkanayhan)
Proposed Solution
We believe the simple first iteration to fix this is as follows:
- Gitaly already has a
bin_dir
config value in our TOML. This should be updated each deploy to<path/that's/there/right/now>/<build_revision>
. - The installer can then place all compiled assets for Gitaly in that directory.
- A HUP is then sent to Gitaly
- After the HUP, the old revision
bin_dirs
directory could be cleaned up.
This should make deploys 'atomic' and allow Gitaly to ship forward incompatible changes. This only requires changes for 14.0 onwards, and involves no Gitaly changes. This does require changes for Omnibus/CNG, anywhere where we support zero downtime deploys.
For more details about this solution, please see the below comment thread