Move pipeline jobs needs to separate query

What does this MR do and why?

This change optimizes how pipeline dependency data is loaded in the pipeline visualization. Previously, the frontend would fetch all job dependency information ("needs") even when viewing the simple stage-based view, which didn't require this data.

The update splits the data fetching into two separate requests: one for basic pipeline details and another specifically for job dependencies. The dependency data is now only requested when users switch to the "layer view" that actually displays these relationships between jobs. The needs data is unlikely to change, so we can avoid polling it.

The code also reorganizes how this dependency data is processed and merged with the main pipeline data, making the system more efficient by avoiding redundant data processing when it's not needed.

References

Screenshots or screen recordings

No visual changes

Screen_Recording_2025-12-09_at_19.43.46

How to set up and validate locally

1. Create a .gitlab-ci.yml config for the pipeline with needs.
2. Navigate to Build -> Pipelines -> New pipeline
3. Verfy the pipeline data is shown correctly for the stage view.
4. Select "job dependencies" and verify the needs data got requested.
5. Toggle "show dependencied" and verify the viewScreen_Recording_2025-12-09_at_19.42.58 shows the needs connections correctly.

Example .gitlab-ci.yml file:

# .gitlab-ci.yml - Mixed pipeline (stages + explicit needs)
stages:
  - prepare
  - build
  - test
  - security
  - package
  - deploy


cache_dependencies:
  stage: prepare
  script:
    - echo "Caching dependencies..."
    - sleep 2

setup_environment:
  stage: prepare
  script:
    - echo "Setting up environment..."
    - sleep 1

validate_config:
  stage: prepare
  script:
    - echo "Validating configuration..."
    - sleep 1


compile_java:
  stage: build
  script:
    - echo "Compiling Java code..."
    - sleep 3

compile_frontend:
  stage: build
  script:
    - echo "Compiling frontend..."
    - sleep 2

# Mixed: Some tests use stages, others use explicit needs
unit_tests:
  stage: test  
  script:
    - echo "Running unit tests..."
    - sleep 2

integration_tests:
  stage: test  
  script:
    - echo "Running integration tests..."
    - sleep 4

frontend_tests:
  stage: test
  needs:
    - compile_frontend
  script:
    - echo "Running frontend tests..."
    - sleep 2

api_tests:
  stage: test
  needs: 
    - compile_java
    - compile_frontend
  script:
    - echo "Running API tests..."
    - sleep 3

# Mixed dependencies
dependency_scan:
  stage: security  
  script:
    - echo "Scanning dependencies..."
    - sleep 2

code_quality:
  stage: security
  needs:
    - compile_java
  script:
    - echo "Running code quality checks..."
    - sleep 3

container_scan:
  stage: security
  needs:
    - unit_tests
    - integration_tests
  script:
    - echo "Scanning container..."
    - sleep 2

# Package stage (stage-based)
build_jar:
  stage: package
  script:
    - echo "Building JAR..."
    - sleep 2

build_docker:
  stage: package
  script:
    - echo "Building Docker image..."
    - sleep 3

# Deploy with mixed dependencies
deploy_staging:
  stage: deploy
  needs: 
    - build_docker
    - dependency_scan
    - code_quality
  script:
    - echo "Deploying to staging..."
    - sleep 2

deploy_docs:
  stage: deploy  
  script:
    - echo "Deploying documentation..."
    - sleep 1

deploy_production:
  stage: deploy
  needs: 
    - deploy_staging
    - container_scan
  script:
    - echo "Deploying to production..."
    - sleep 2
  when: manual

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Related to #549069