The `ci_composite_status` and `ci_atomic_processing` fails for some workflows

Due to awesome work of @furkanayhan on !29405 (merged), we discovered some differences in behaviour.

We very freely configure different feature flags:

  • ci_composite_status is an efficient way to calculate CI status in Ruby
  • ci_atomic_processing is an efficient way to update Pipeline/Stage/Build atomically

These two feature flags are distinct, and ci_composite_status does affect system behavior if ci_atomic_processing=false is used. It switches LegacyProcessingService to use a composite status calculation.

There are a few cases that are broken today that makes system to behave differently to ci_composite_status=false AND ci_atomic_processing=false. The idea behind Atomic Processing is that it does not change the system behaviour, or we do it very carefully.

1. Failures of ci_composite_status=true and ci_atomic_processing=false

1.1. dag_build_allow_failure_test_on_failure.yml

```yaml config: build: stage: build allow_failure: true script: exit 1 
  test:
    stage: test
    when: on_failure
    script: exit 0
    needs: [build]

  deploy:
    stage: deploy
    script: exit 0
    needs: [test]

init:
  expect:
    pipeline: pending
    stages:
      build: pending # this is created for that test-case
      test: created
      deploy: created
    jobs:
      build: pending
      test: created
      deploy: created

transitions:
  - event: drop
    jobs: [build]
    expect:
      pipeline: success
      stages:
        build: success
        test: skipped
        deploy: skipped
      jobs:
        build: failed
        test: skipped
        deploy: skipped
```

Result

 "jobs" => {"build"=>"pending", "deploy"=>"created", "test"=>"created"},
 "pipeline" => "pending",
-"stages" => {"build"=>"pending", "deploy"=>"created", "test"=>"created"},
+"stages" => {"build"=>"created", "deploy"=>"created", "test"=>"created"},

1.2. dag_builds_succeed_test_on_failure_deploy_always.yml

```yaml config: build_1: stage: build script: exit 0 
  build_2:
    stage: build
    script: exit 0

  test:
    stage: test
    script: exit 0
    when: on_failure

  deploy:
    stage: deploy
    script: exit 0
    when: always
    needs: [build_1, test]

init:
  expect:
    pipeline: pending
    stages:
      build: pending
      test: created
      deploy: created
    jobs:
      build_1: pending
      build_2: pending
      test: created
      deploy: created

transitions:
  - event: success
    jobs: [build_1, build_2]
    expect:
      pipeline: running
      stages:
        build: success
        test: skipped
        deploy: pending
      jobs:
        build_1: success
        build_2: success
        test: skipped
        deploy: pending

  - event: success
    jobs: [deploy]
    expect:
      pipeline: success
      stages:
        build: success
        test: skipped
        deploy: success
      jobs:
        build_1: success
        build_2: success
        test: skipped
        deploy: success
```

Result

 "jobs" => {"build_1"=>"success", "build_2"=>"success", "deploy"=>"pending", "test"=>"skipped"},
 "pipeline" => "running",
-"stages" => {"build"=>"success", "deploy"=>"pending", "test"=>"skipped"},
+"stages" => {"build"=>"success", "deploy"=>"created", "test"=>"skipped"},

2. Failures of ci_composite_status=true and ci_atomic_processing=true

2.1. dag_test_manual_allow_failure_true_other_test_succeeds.yml

```yaml config: test1: stage: test script: exit 0 
  test2:
    stage: test
    when: manual
    allow_failure: true
    script: exit 1

  deploy:
    stage: deploy
    script: exit 0
    needs: [test1, test2]

init:
  expect:
    pipeline: pending
    stages:
      test: pending
      deploy: created
    jobs:
      test1: pending
      test2: manual
      deploy: created

transitions:
  - event: success
    jobs: [test1]
    expect:
      pipeline: running
      stages:
        test: success
        deploy: created
      jobs:
        test1: success
        test2: manual
        deploy: created
```

Result

-"jobs" => {"deploy"=>"created", "test1"=>"success", "test2"=>"manual"},
+"jobs" => {"deploy"=>"pending", "test1"=>"success", "test2"=>"manual"},
 "pipeline" => "running",
-"stages" => {"deploy"=>"created", "test"=>"success"},
+"stages" => {"deploy"=>"pending", "test"=>"success"},

2.2. dag_test_manual_allow_failure_true_deploy_always.yml

```yaml config: test: stage: test when: manual allow_failure: true script: exit 1 
  deploy:
    stage: deploy
    when: always
    script: exit 0
    needs: [test]

init:
  expect:
    pipeline: created
    stages:
      test: skipped
      deploy: created
    jobs:
      test: manual
      deploy: created

transitions: []
```

Result

-"jobs" => {"deploy"=>"created", "test"=>"manual"},
-"pipeline" => "created",
-"stages" => {"deploy"=>"created", "test"=>"skipped"},
+"jobs" => {"deploy"=>"pending", "test"=>"manual"},
+"pipeline" => "pending",
+"stages" => {"deploy"=>"pending", "test"=>"skipped"},

2.3. dag_test_manual_allow_failure_true_deploy_on_failure.yml

```yaml config: test: stage: test when: manual allow_failure: true script: exit 1 
  deploy:
    stage: deploy
    when: on_failure
    script: exit 0
    needs: [test]

init:
  expect:
    pipeline: created
    stages:
      test: skipped
      deploy: created
    jobs:
      test: manual
      deploy: created

transitions:
  - event: enqueue
    jobs: [test]
    expect:
      pipeline: pending
      stages:
        test: running
        deploy: created
      jobs:
        test: pending
        deploy: created

  - event: drop
    jobs: [test]
    expect:
      pipeline: success
      stages:
        test: success
        deploy: skipped
      jobs:
        test: failed
        deploy: skipped
```

Result

-"jobs" => {"deploy"=>"created", "test"=>"manual"},
-"pipeline" => "created",
-"stages" => {"deploy"=>"created", "test"=>"skipped"},
+"jobs" => {"deploy"=>"skipped", "test"=>"manual"},
+"pipeline" => "skipped",
+"stages" => {"deploy"=>"skipped", "test"=>"skipped"},

2.4. dag_test_manual_allow_failure_true.yml

```yaml config: test: stage: test when: manual allow_failure: true script: exit 1 
  deploy:
    stage: deploy
    script: exit 0
    needs: [test]

init:
  expect:
    pipeline: created
    stages:
      test: skipped
      deploy: created
    jobs:
      test: manual
      deploy: created

transitions:
  - event: enqueue
    jobs: [test]
    expect:
      pipeline: pending
      stages:
        test: running
        deploy: created
      jobs:
        test: pending
        deploy: created

  - event: run
    jobs: [test]
    expect:
      pipeline: running
      stages:
        test: running
        deploy: created
      jobs:
        test: running
        deploy: created

  - event: drop
    jobs: [test]
    expect:
      pipeline: running
      stages:
        test: success
        deploy: pending
      jobs:
        test: failed
        deploy: pending
```

Result

-"jobs" => {"deploy"=>"created", "test"=>"manual"},
-"pipeline" => "created",
-"stages" => {"deploy"=>"created", "test"=>"skipped"},
+"jobs" => {"deploy"=>"skipped", "test"=>"manual"},
+"pipeline" => "skipped",
+"stages" => {"deploy"=>"skipped", "test"=>"skipped"},
Edited by Kamil Trzciński