Increase in pipeline ref creation failure from 10:commit: verify file system operations: read: path was modified after read error

We are seeing an uptick of whole CI pipeline failures due to a pipeline ref not created before the runner starts a job. Sample pipeline: https://gitlab.com/gitlab-org/gitlab/-/pipelines/1821480587

Each job fails with something like:

Initialized empty Git repository in /builds/gitlab-org/gitlab/.git/
Created fresh repository.
fatal: couldn't find remote ref refs/pipelines/1821480587

If we look at the Sentry logs in https://new-sentry.gitlab.net/organizations/gitlab/issues/1521340/?project=3&query=is%3Aunresolved+correlation_id%3A01JVCS2Z586DF52K2H3GKMAGB2&referrer=issue-stream&statsPeriod=14d&stream_index=0, we see:

Gitlab::Git::CommandError

10:commit: verify file system operations: read: path was modified after read.

This error comes from ReadWriteConflictError: https://gitlab.com/gitlab-org/gitaly/-/blob/c4db6ec453df2c6b68d538ff07486e8d8554f4ae/internal/gitaly/storage/storagemgr/partition/conflict/fshistory/errors.go#L17-19

There are really only two places this is created:

% git grep NewReadWriteConflictError | grep -v test
internal/gitaly/storage/storagemgr/partition/conflict/fshistory/errors.go:// NewReadWriteConflictError returns an error detailing a conflicting read.
internal/gitaly/storage/storagemgr/partition/conflict/fshistory/errors.go:func NewReadWriteConflictError(path string, readLSN, writeLSN storage.LSN) error {
internal/gitaly/storage/storagemgr/partition/conflict/fshistory/history.go:		return NewReadWriteConflictError(path, tx.readLSN, node.writeLSN)
internal/gitaly/storage/storagemgr/partition/conflict/fshistory/tree.go:			return nil, NewReadWriteConflictError(currentPath, tx.readLSN, child.writeLSN)```

@engwan wondered if we're seeing an uptick of failures due to gitlab!187651 (merged).

@samihiltunen This error looks directly related to Gitaly transactions, however.