Geo: Excessive Gitaly error responses due to pipeline ref can't be created yet

Problem

When a pipeline_ref_created event is replicated on a secondary Geo site, if the target revision doesn't exist yet, then Gitaly responds with an error when attempting to create the ref. The Sidekiq job does not fail since the error is rescued. But this is causing high Gitaly error rates and excessive unnecessary alerts for GitLab Dedicated. I suspect this may be exacerbated during backfill, or when there is a large, highly active repo.

extra.sidekiq.args: pipeline, pipeline_ref_created, {"model_record_id"=>4990768}
extra.sidekiq.class: Geo::EventWorker
meta.feature_category: geo_replication
exception.message: 13:resolving new revision: reference not found.

lib/gitlab/git/wraps_gitaly_errors.rb:24:in `rescue in wrapped_gitaly_errors',
lib/gitlab/git/wraps_gitaly_errors.rb:6:in `wrapped_gitaly_errors',
lib/gitlab/git/repository.rb:882:in `write_ref',
app/models/repository.rb:1041:in `create_ref',
app/models/ci/persistent_ref.rb:14:in `create_ref',
app/models/ci/persistent_ref.rb:28:in `create',
app/models/ci/pipeline.rb:1306:in `ensure_persistent_ref',
ee/app/models/ee/ci/pipeline.rb:211:in `ensure_persistent_ref',
ee/app/replicators/geo/pipeline_replicator.rb:18:in `consume_event_pipeline_ref_created',
ee/lib/gitlab/geo/replicator.rb:269:in `consume',
ee/app/services/geo/event_service.rb:18:in `execute',
ee/app/workers/geo/event_worker.rb:15:in `perform',
lib/gitlab/sidekiq_middleware/skip_jobs.rb:49:in `call',
lib/gitlab/database/load_balancing/sidekiq_server_middleware.rb:29:in `call',
lib/gitlab/sidekiq_middleware/duplicate_jobs/strategies/until_executing.rb:16:in `perform',
lib/gitlab/sidekiq_middleware/duplicate_jobs/duplicate_job.rb:44:in `perform',
lib/gitlab/sidekiq_middleware/duplicate_jobs/server.rb:8:in `call',
lib/gitlab/sidekiq_middleware/pause_control/strategies/base.rb:31:in `perform',
lib/gitlab/sidekiq_middleware/pause_control/strategy_handler.rb:22:in `perform',
lib/gitlab/sidekiq_middleware/pause_control/server.rb:8:in `call',
lib/gitlab/sidekiq_middleware/worker_context.rb:9:in `wrap_in_optional_context',
lib/gitlab/sidekiq_middleware/worker_context/server.rb:19:in `block in call',
lib/gitlab/application_context.rb:124:in `block in use',
lib/gitlab/application_context.rb:124:in `use',
lib/gitlab/application_context.rb:62:in `with_context',
lib/gitlab/sidekiq_middleware/worker_context/server.rb:17:in `call',
lib/gitlab/sidekiq_status/server_middleware.rb:7:in `call',
lib/gitlab/sidekiq_versioning/middleware.rb:9:in `call',
lib/gitlab/sidekiq_middleware/query_analyzer.rb:7:in `block in call',
lib/gitlab/database/query_analyzer.rb:37:in `within',
lib/gitlab/sidekiq_middleware/query_analyzer.rb:7:in `call',
lib/gitlab/sidekiq_middleware/admin_mode/server.rb:14:in `call',
lib/gitlab/sidekiq_middleware/instrumentation_logger.rb:9:in `call',
lib/gitlab/sidekiq_middleware/batch_loader.rb:7:in `call',
lib/gitlab/sidekiq_middleware/extra_done_log_metadata.rb:7:in `call',
lib/gitlab/sidekiq_middleware/request_store_middleware.rb:8:in `block in call',
lib/gitlab/sidekiq_middleware/request_store_middleware.rb:7:in `call',
lib/gitlab/sidekiq_middleware/server_metrics.rb:94:in `block in call',
lib/gitlab/sidekiq_middleware/server_metrics.rb:122:in `block in instrument',
lib/gitlab/metrics/background_transaction.rb:33:in `run',
lib/gitlab/sidekiq_middleware/server_metrics.rb:122:in `instrument',
lib/gitlab/sidekiq_middleware/server_metrics.rb:93:in `call',
lib/gitlab/sidekiq_middleware/monitor.rb:10:in `block in call',
lib/gitlab/sidekiq_daemon/monitor.rb:46:in `within_job',
lib/gitlab/sidekiq_middleware/monitor.rb:9:in `call',
lib/gitlab/sidekiq_middleware/size_limiter/server.rb:13:in `call',
lib/gitlab/sidekiq_logging/structured_logger.rb:21:in `call'