NoMethodError: undefined method `parent_ids' for nil:NilClass // Pipelines are not running automatically anymore

Summary

our self-hosted instance have thousands of dead jobs in sidekiq and pipelines are not running automatic anymore. Also emails related to pipelines cannot be send out anymore.

Steps to reproduce

this is only happening in our live environment. test environment with same software version is working fine. this happens for alle projects with pipelines on the environment.

What is the current bug behavior?

Job MergeRequestMergeabilityCheckWorker receive this error: "NoMethodError: undefined method parent_ids' for nil:NilClass" Job Notify#pipeline_failed_email receive this error: ActionView::Template::Error: undefined method truncate' for nil:NilClass

What is the expected correct behavior?

Pipelines can run automatically and emails are sent out.

Relevant logs

{"severity":"WARN","time":"2022-07-04T02:10:23.596Z","retry":3,"queue":"merge_request_mergeability_check","version":0,"args":["6424"],"class":"MergeRequestMergeabilityCheckWorker","jid":"b2a7c4a25df5e05efc20ff33","created_at":"2022-07-04T02:10:23.319Z","correlation_id":"01G73FJ6RN666YJPNP6J25XWMF","meta.user":"fj1613bv","meta.project":"*************","meta.root_namespace":"**************","meta.client_id":"user/4741","meta.caller_id":"Projects::MergeRequestsController#show","meta.remote_ip":"208.127.121.242","meta.feature_category":"code_review","worker_data_consistency":"always","idempotency_key":"resque:gitlab:duplicate:merge_request_mergeability_check:c3f861dfc0de1cb2cddea335b30ad7507704f9566ee02d529e50947146136529","size_limiter":"validated","enqueued_at":"2022-07-04T02:10:23.322Z","job_size_bytes":6,"pid":813979,"message":"MergeRequestMergeabilityCheckWorker JID-b2a7c4a25df5e05efc20ff33: fail: 0.27127 sec","job_status":"fail","scheduling_latency_s":0.00222,"gitaly_calls":1,"gitaly_duration_s":0.111731,"rugged_calls":1,"rugged_duration_s":0.009856,"redis_calls":6,"redis_duration_s":0.052844999999999996,"redis_read_bytes":357,"redis_write_bytes":1387,"redis_cache_calls":4,"redis_cache_duration_s":0.047053,"redis_cache_read_bytes":346,"redis_cache_write_bytes":807,"redis_queues_read_bytes":9,"redis_queues_write_bytes":299,"redis_shared_state_calls":2,"redis_shared_state_duration_s":0.005792,"redis_shared_state_read_bytes":2,"redis_shared_state_write_bytes":281,"db_count":11,"db_write_count":0,"db_cached_count":2,"db_replica_count":0,"db_primary_count":11,"db_main_count":11,"db_main_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":2,"db_main_cached_count":2,"db_main_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_main_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.022,"db_main_duration_s":0.022,"db_main_replica_duration_s":0.0,"cpu_s":0.056746,"mem_objects":9587,"mem_bytes":1544696,"mem_mallocs":2633,"mem_total_bytes":1928176,"duration_s":0.27127,"completed_at":"2022-07-04T02:10:23.595Z","load_balancing_strategy":"primary","error_message":"undefined method `parent_ids' for nil:NilClass","error_class":"NoMethodError","error_backtrace":["app/services/merge_requests/merge_to_ref_service.rb:26:in `execute'","app/services/merge_requests/mergeability_check_service.rb:160:in `merge_to_ref'","app/services/merge_requests/mergeability_check_service.rb:115:in `update_merge_status'","app/services/merge_requests/mergeability_check_service.rb:60:in `check_mergeability'","app/services/merge_requests/mergeability_check_service.rb:48:in `block in execute'","lib/gitlab/exclusive_lease_helpers.rb:38:in `in_lock'","app/services/merge_requests/mergeability_check_service.rb:86:in `in_write_lock'","app/services/merge_requests/mergeability_check_service.rb:42:in `execute'","app/workers/merge_request_mergeability_check_worker.rb:28:in `perform'","lib/gitlab/database/load_balancing/sidekiq_server_middleware.rb:26:in `call'","lib/gitlab/sidekiq_middleware/duplicate_jobs/strategies/until_executing.rb:16:in `perform'","lib/gitlab/sidekiq_middleware/duplicate_jobs/duplicate_job.rb:58:in `perform'","lib/gitlab/sidekiq_middleware/duplicate_jobs/server.rb:8:in `call'","lib/gitlab/sidekiq_middleware/worker_context.rb:9:in `wrap_in_optional_context'","lib/gitlab/sidekiq_middleware/worker_context/server.rb:17:in `block in call'","lib/gitlab/application_context.rb:93:in `block in use'","lib/gitlab/application_context.rb:93:in `use'","lib/gitlab/application_context.rb:44:in `with_context'","lib/gitlab/sidekiq_middleware/worker_context/server.rb:15:in `call'","lib/gitlab/sidekiq_status/server_middleware.rb:7:in `call'","lib/gitlab/sidekiq_versioning/middleware.rb:9:in `call'","lib/gitlab/sidekiq_middleware/query_analyzer.rb:7:in `block in call'","lib/gitlab/database/query_analyzer.rb:46:in `within'","lib/gitlab/sidekiq_middleware/query_analyzer.rb:7:in `call'","lib/gitlab/sidekiq_middleware/admin_mode/server.rb:14:in `call'","lib/gitlab/sidekiq_middleware/instrumentation_logger.rb:9:in `call'","lib/gitlab/sidekiq_middleware/batch_loader.rb:7:in `call'","lib/gitlab/sidekiq_middleware/extra_done_log_metadata.rb:7:in `call'","lib/gitlab/sidekiq_middleware/request_store_middleware.rb:10:in `block in call'","lib/gitlab/with_request_store.rb:17:in `enabling_request_store'","lib/gitlab/with_request_store.rb:10:in `with_request_store'","lib/gitlab/sidekiq_middleware/request_store_middleware.rb:9:in `call'","lib/gitlab/sidekiq_middleware/server_metrics.rb:74:in `block in call'","lib/gitlab/sidekiq_middleware/server_metrics.rb:97:in `block in instrument'","lib/gitlab/metrics/background_transaction.rb:33:in `run'","lib/gitlab/sidekiq_middleware/server_metrics.rb:97:in `instrument'","lib/gitlab/sidekiq_middleware/server_metrics.rb:73:in `call'","lib/gitlab/sidekiq_middleware/monitor.rb:10:in `block in call'","lib/gitlab/sidekiq_daemon/monitor.rb:49:in `within_job'","lib/gitlab/sidekiq_middleware/monitor.rb:9:in `call'","lib/gitlab/sidekiq_middleware/size_limiter/server.rb:13:in `call'","lib/gitlab/sidekiq_logging/structured_logger.rb:21:in `call'"],"db_duration_s":0.021553}

Details of package version

Gitlab 14.10.5-ee GitLab Shell 13.25.1 GitLab Workhorse v14.10.5 GitLab AP Iv4 GitLab Pages 1.56.1 GitLab KAS 14.10.0 Ruby 2.7.5p203 Rails 6.1.4.7 PostgreSQ L13.6 Redis 6.0.14

Environment details

omnibus installation with external db + external redis + external gitaly + external object storage

Configuration details

Provide the relevant sections of `/etc/gitlab/gitlab.rb`

Please let me know if you need something specific, our configuration is very big because of all this external services.

Any other information that is required can be added if requested.