Backend: Scheduled pipelines are not triggering anymore when using rules

Summary

Scheduled pipelines for some of my project have not triggering anymore for 3 months in my Omnibus installation.

Note: I checked all similar issues with pipelines not triggering but the cause seems to be something different here.

GitLab version: 12.9.5 (Omnibus installation)

What is the current bug behavior?

In the screenshot below the "Hourly execution (new)" scheduled pipeline was created just half an hour before that run. I looked in /var/log/gitlab/gitlab-rails/sidekiq.log and combined log at the scheduled time of execution but could not find anything.

What is the expected correct behavior?

The pipelines are triggered as scheduled

Example Project

From #325459 (closed) - https://gitlab.com/brett.higgins/scheduled-pipeline-bug-demo

Relevant logs and/or screenshots

This is the .gitlab-ci.yml for the project mentioned:

stages:
  - build
  - run

build-image:
  stage: build
  image:
    name: gcr.io/kaniko-project/executor:debug
    entrypoint: [""]
  script:
    - echo "{\"auths\":{\"$CI_REGISTRY\":{\"username\":\"$CI_DOCKER_REGISTRY_USER\",\"password\":\"$CI_DOCKER_REGISTRY_PASSWORD\"}}}" > /kaniko/.docker/config.json
    - /kaniko/executor --context $CI_PROJECT_DIR --dockerfile $CI_PROJECT_DIR/Dockerfile --destination $CI_REGISTRY_IMAGE
  rules:
    - if: '$CI_COMMIT_BRANCH == "master"'
      when: always

sync-groups:
  stage: run
  image: $CI_REGISTRY_IMAGE
  script:
    - echo "Testing sync-groups
  rules:
    - if: '$CI_PIPELINE_SOURCE == "schedule"'
      when: always