FF_NETWORK_PER_BUILD configuration in variables is lost on saas-linux-medium-amd64 runners

Summary

Feature flag FF_NETWORK_PER_BUILD configured in variables can not be enabled on saas-linux-medium-amd64 runners. However other feature flags work correctly, for example: FF_USE_FASTZIP. Small and large runners are not affected.

Steps to reproduce

Confligure FF_NETWORK_PER_BUILD in .gitlab-ci.yml, either for the whole pipeline or just for a given job. The bug is reproducible with both.

.gitlab-ci.yml

---
stages:
  - build
  - test
  - deploy

build-pypi-package:
  stage: build
  image: python:3.9
  before_script:
    - pip install --upgrade pip
    - pip install --upgrade build
  script:
    - python -m build
  artifacts:
    expire_in: 1 day
    paths:
      - dist/

run-test:
  stage: test
  tags: [saas-linux-medium-amd64]
  parallel:
    matrix:
      # pin conda:3.7-15 due to #1829
      - SARDANA_IMAGE: [":latest", "/conda:3.7-15", "/conda:3.8", "/conda:3.9", "/conda:3.10", "/conda:3.11"]
  image: registry.gitlab.com/sardana-org/sardana-docker${SARDANA_IMAGE}
  variables:
    GIT_STRATEGY: none
    MYSQL_HOST: "gitlab-runner-tango-mysql:3306"
    MYSQL_DATABASE: "tango"
    MYSQL_ROOT_PASSWORD: "root"
    MYSQL_USER: "tango"
    MYSQL_PASSWORD: "tango"
    # Required for communication between services to be enabled
    FF_NETWORK_PER_BUILD: "true"
  services:
    - name: registry.gitlab.com/tango-controls/docker/mysql:5
      alias: gitlab-runner-tango-mysql
    - name: registry.gitlab.com/tango-controls/docker/tango-db
      alias: gitlab-runner-tango-db
  before_script:
    # Replace the sardana version installed in the image with the local one
    - python3 -m pip uninstall -y sardana || true
    - WHEEL_NAME=dist/sardana-*.whl
    - python3 -m pip install $WHEEL_NAME
    # Install config dependencies, not really sure how to do it better using
    # whl file expansion (GLOB) and square brackets (GLOB character):
    - CONFIG_EXTRAS="[config]"
    - WHEEL_NAME_WITH_CONFIG_EXTRAS="$(printf '%s%s' $WHEEL_NAME "$CONFIG_EXTRAS")"
    - python3 -m pip install "$WHEEL_NAME_WITH_CONFIG_EXTRAS"
    # Use IP address for TANGO_HOST to avoid slow DNS issue in gitlab-ci
    - export TANGO_HOST=$(python -c "import socket;print(socket.gethostbyname('gitlab-runner-tango-db'))"):10000
    - echo Waiting for Tango services to be ready...
    - sleep 10
  script:
    - INST_DIR=$(python3 -c 'import sardana; print(sardana.__file__.rsplit("/",1)[0])')
    # pytest-xvfb shall be part of the image
    - pytest --junitxml=report.xml $INST_DIR
  after_script:
    - mkdir -p logs
    - cp -a /tmp/tango-mambauser logs/
  artifacts:
    when: always
    paths:
      - logs
    reports:
      junit: report.xml

build-doc:
  stage: test
  image: registry.gitlab.com/sardana-org/sardana-docker/doc
  script:
    - xvfb-run sphinx-build -W doc/source/ public
    - echo "Documentation can be found at https://$CI_PROJECT_NAMESPACE.gitlab.io/-/$CI_PROJECT_NAME/-/jobs/$CI_JOB_ID/artifacts/public/index.html"
  artifacts:
    paths:
      - public
  environment:
    name: Docs-dev
    url: "https://$CI_PROJECT_NAMESPACE.gitlab.io/-/$CI_PROJECT_NAME/-/jobs/$CI_JOB_ID/artifacts/public/index.html"

pages:
  stage: deploy
  image: alpine
  variables:
    GIT_STRATEGY: none
  script:
    - echo "Deploying already-built docs"
  artifacts:
    paths:
      - public
  rules:
    - if: '$CI_COMMIT_TAG =~ /^[0-9]+\.[0-9]+\.[0-9]+.*$/'

release-pypi-package:
  stage: deploy
  image: python:3.9
  variables:
    GIT_STRATEGY: none
  before_script:
    - pip install twine
  script:
    - twine upload dist/*
  rules:
    - if: '$CI_COMMIT_TAG =~ /^[0-9]+\.[0-9]+\.[0-9]+.*$/'

Actual behavior

The flag does not appear on the job startup logs:

Expected behavior

The flag appear on th job startup logs and gets enabled

Relevant logs and/or screenshots

Last successful job execution from 29-03-2023 initial logs:

Running with gitlab-runner 15.9.0~beta.115.g598a7c91 (598a7c91)
  on green-4.saas-linux-medium-amd64.runners-manager.gitlab.com/default Yy-Tzs3Q, system ID: s_57a32fe7e1c1
  feature flags: FF_NETWORK_PER_BUILD:true, FF_USE_IMPROVED_URL_MASKING:true. `FF_NETWORK_PER_BUILD` works correctly on small and large runners.

An example of a failed job execution from 30-03-2023 initial logs:

Running with gitlab-runner 15.9.0~beta.115.g598a7c91 (598a7c91)
  on green-4.saas-linux-medium-amd64.runners-manager.gitlab.com/default Yy-Tzs3Q, system ID: s_57a32fe7e1c1
  feature flags: FF_USE_IMPROVED_URL_MASKING:true

Environment description

Using shared Runners on GitLab.com: saas-linux-medium-amd64

Used GitLab Runner version

15.9.0~beta.115.g598a7c91 (598a7c91)