We are seeing an uptick of whole CI pipeline failures due to a pipeline ref not created before the runner starts a job. Sample pipeline: https://gitlab.com/gitlab-org/gitlab/-/pipelines/1821480587 Each job fails with something like: ``` Initialized empty Git repository in /builds/gitlab-org/gitlab/.git/ Created fresh repository. fatal: couldn't find remote ref refs/pipelines/1821480587 ``` If we look at the Sentry logs in https://new-sentry.gitlab.net/organizations/gitlab/issues/1521340/?project=3&query=is%3Aunresolved+correlation_id%3A01JVCS2Z586DF52K2H3GKMAGB2&referrer=issue-stream&statsPeriod=14d&stream_index=0, we see: ``` Gitlab::Git::CommandError 10:commit: verify file system operations: read: path was modified after read. ``` This error comes from `ReadWriteConflictError`: https://gitlab.com/gitlab-org/gitaly/-/blob/c4db6ec453df2c6b68d538ff07486e8d8554f4ae/internal/gitaly/storage/storagemgr/partition/conflict/fshistory/errors.go#L17-19 There are really only two places this is created: ``` % git grep NewReadWriteConflictError | grep -v test internal/gitaly/storage/storagemgr/partition/conflict/fshistory/errors.go:// NewReadWriteConflictError returns an error detailing a conflicting read. internal/gitaly/storage/storagemgr/partition/conflict/fshistory/errors.go:func NewReadWriteConflictError(path string, readLSN, writeLSN storage.LSN) error { internal/gitaly/storage/storagemgr/partition/conflict/fshistory/history.go: return NewReadWriteConflictError(path, tx.readLSN, node.writeLSN) internal/gitaly/storage/storagemgr/partition/conflict/fshistory/tree.go: return nil, NewReadWriteConflictError(currentPath, tx.readLSN, child.writeLSN)``` ``` @engwan wondered if we're seeing an uptick of failures due to https://gitlab.com/gitlab-org/gitlab/-/merge_requests/187651. @samihiltunen This error looks directly related to Gitaly transactions, however.