Calculate previous stage jobs on the frontend

What does this MR do and why?

This code change simplifies how CI/CD pipeline job dependencies are handled. Previously, the system tracked two separate types of job dependencies: explicit "needs" (jobs that must complete before this job runs) and "previous stage jobs" (all jobs from the previous pipeline stage).

The update removes the separate tracking of "previous stage jobs" and instead automatically includes all jobs from the previous stage when calculating dependencies. This means jobs now inherit dependencies from the entire previous stage by default, while still allowing explicit dependencies to be specified. The change also fixes some circular dependency issues and improves performance by reducing duplicate data processing.

The main benefits are cleaner code, better handling of pipeline job relationships, and more predictable dependency behavior for users setting up their CI/CD pipelines.

References

Screenshots or screen recordings

No visual changes

Before	After

How to set up and validate locally

1. Create a setup for a pipeline with needs
2. Visit Build -> Pipelines page
3. Run the pipeline and visit the details page
4. Verify the job dependency is showing correctly.

Example .gitlab-ci.yml config:

# .gitlab-ci.yml - Mixed pipeline (stages + explicit needs)
stages:
  - prepare
  - build
  - test
  - security
  - package
  - deploy


cache_dependencies:
  stage: prepare
  script:
    - echo "Caching dependencies..."
    - sleep 2

setup_environment:
  stage: prepare
  script:
    - echo "Setting up environment..."
    - sleep 1

validate_config:
  stage: prepare
  script:
    - echo "Validating configuration..."
    - sleep 1


compile_java:
  stage: build
  script:
    - echo "Compiling Java code..."
    - sleep 3

compile_frontend:
  stage: build
  script:
    - echo "Compiling frontend..."
    - sleep 2

# Mixed: Some tests use stages, others use explicit needs
unit_tests:
  stage: test  
  script:
    - echo "Running unit tests..."
    - sleep 2

integration_tests:
  stage: test  
  script:
    - echo "Running integration tests..."
    - sleep 4

frontend_tests:
  stage: test
  needs:
    - compile_frontend
  script:
    - echo "Running frontend tests..."
    - sleep 2

api_tests:
  stage: test
  needs: 
    - compile_java
    - compile_frontend
  script:
    - echo "Running API tests..."
    - sleep 3

# Mixed dependencies
dependency_scan:
  stage: security  
  script:
    - echo "Scanning dependencies..."
    - sleep 2

code_quality:
  stage: security
  needs:
    - compile_java
  script:
    - echo "Running code quality checks..."
    - sleep 3

container_scan:
  stage: security
  needs:
    - unit_tests
    - integration_tests
  script:
    - echo "Scanning container..."
    - sleep 2

# Package stage (stage-based)
build_jar:
  stage: package
  script:
    - echo "Building JAR..."
    - sleep 2

build_docker:
  stage: package
  script:
    - echo "Building Docker image..."
    - sleep 3

# Deploy with mixed dependencies
deploy_staging:
  stage: deploy
  needs: 
    - build_docker
    - dependency_scan
    - code_quality
  script:
    - echo "Deploying to staging..."
    - sleep 2

deploy_docs:
  stage: deploy  
  script:
    - echo "Deploying documentation..."
    - sleep 1

deploy_production:
  stage: deploy
  needs: 
    - deploy_staging
    - container_scan
  script:
    - echo "Deploying to production..."
    - sleep 2
  when: manual

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Related to #549069