Private dotenv artifacts not accessible to downstream jobs

Summary

Linked Issue: https://gitlab.com/gitlab-org/security/gitlab/-/issues/1139

CI/CD pipelines can generate dotenv artifacts to pass variables to subsequent jobs in the same pipeline. When the artifact has public: false set subsequent jobs in the same project and pipeline no longer load variables from the dotenv.

Steps to reproduce

1. Create a project
2. Set up CI/CD:

CI YAML example

stages:
  - artifact
  - use_artifact

image: alpine:latest

create-artifact:
  stage: artifact
  script:
    - | 
      echo VARIABLE=12345 >> data.env
      echo OTHER_VARIABLE=this-is-a-key >> data.env
      echo VERSION_VARIABLE=1.2.3.4 >> data.env
      echo VERSION_TAG=1.2.3.${CI_PIPELINE_ID} >> data.env
      echo "SOME_PATHS='/mnt/stars/allsky/ccd_54528347-52c5-49e9-9fe2-65c3a007035f/20240508/day/08_05/ccd1_20240508_055734.jpg,/mnt/stars/allsky/ccd_54528347-52c5-49e9-9fe2-65c3a007035f/20240508/day/08_05/ccd1_20240508_055749.jpg'" >> data.env
  artifacts:
    public: false
    reports:
      dotenv: data.env

use-artifact:
  stage: use_artifact
  script:
    - echo -e "Checking variables from last job:\n VARIABLE - $VARIABLE \n OTHER_VARIABLE - $OTHER_VARIABLE \n VERSION_VARIABLE - $VERSION_VARIABLE \n VERSION_TAG - $VERSION_TAG \n SOME_PATHS - $SOME_PATHS"

3. Run pipeline and observe that variables are not populated as expected.

Example Project

https://gitlab.com/duncan_harris_ultimate_group/dotenv-543743

This example project has a branch with public: true where the pipeline works as documented.

What is the current bug behavior?

Jobs in the same pipeline are not able to access non-public dotenv artifacts from prior job stages

What is the expected correct behavior?

Downstream jobs in the same pipeline should have access to dotenv artifacts

Relevant logs and/or screenshots

$ echo -e "Checking variables from last job:\n VARIABLE - $VARIABLE \n OTHER_VARIABLE - $OTHER_VARIABLE \n VERSION_VARIABLE - $VERSION_VARIABLE \n VERSION_TAG - $VERSION_TAG \n SOME_PATHS - $SOME_PATHS"
Checking variables from last job:
 VARIABLE -  
 OTHER_VARIABLE -  
 VERSION_VARIABLE -  
 VERSION_TAG - 
 SOME_PATHS - 

Output of checks