## Summary Replication of Pipeline Artifacts are failing to sync on a Geo secondary site, the only error logged is the following which we suspect is hiding the root cause: ``` Error while attempting to sync: undefined method `log_error' for an instance of Gitlab::Geo::Replication::BlobDownloader ``` ## Steps to Reproduce TBC ## Expected Behavior Pipeline Artifacts sync successfully to the Geo secondary site without errors. If an error occurs it is logged correctly and usefully to help resolve. ## Actual Behavior 477 Pipeline Artifacts fail to sync with: ``` Error while attempting to sync: undefined method `log_error' for an instance of Gitlab::Geo::Replication::BlobDownloader ``` ## Root Cause Analysis (generated by Duo) `Gitlab::Geo::Replication::BlobDownloader` (`ee/lib/gitlab/geo/replication/blob_downloader.rb`) includes `EE::GeoHelper` (line 7) and calls `log_error` (line 149, inside `ensure_destination_path_exists`). The `NoMethodError` indicates that `log_error` is not available on the `BlobDownloader` instance, suggesting that `EE::GeoHelper` is either not providing this method in the affected version, or the logging concern has been moved/removed without updating `BlobDownloader`. A similar historical precedent exists: `NoMethodError: undefined method 'log_info' for Geo::RepositorySyncWorker` was fixed in gitlab-org/gitlab-ee!5223 by ensuring the correct logging mixin was included. ## Impact - 477 Pipeline Artifacts failing to replicate to the Geo secondary site - Geo replication is incomplete, posing a risk to disaster recovery readiness - It was not possible to diagnose the root issue that caused the failure as there was nothing useful logged. ## Environment - GitLab installation type: Omnibus - GitLab version: On-prem Primary 18.8.3, Cloud hosted Secondary 18.10.3-ee - to facilitate migration - DB: Primary uses onboard DB, secondary uses dedicated external PostgreSQL - both 16 - Geo role: Secondary site - Storage: Primary and secondary share the same S3 object storage bucket for blobs; on the secondary ## Workaround None known at this time. A workaround to identify the problem in this particular case was to connect to the Rails Console, define a method called `log_error` and execute the replication again. This was the code that was used ``` ::Gitlab::Geo::Replication::BlobDownloader.define_method(:log_error) { |*args, **kwargs| puts args; puts kwargs; nil } registry = ::Geo::PipelineArtifactRegistry.failed.last ::Gitlab::Geo::Replication::BlobDownloader.new(replicator: registry.replicator).execute ``` The `PipelineArtifactRegistry` may need to change depending upon the Artifacts that failed. In this case, when we executed the job again, we could see the root cause was due to incorrect ownership of the directory to which we were attempting to download. ## Labels `group::geo` `type::bug`