Skip to content

API Pagination on fetch_mr_plan job (max 20)

Summary

My team and I are currently working on automating applying tofu in all our client environments, we have around ~30 directories that correspond to the client environment.

We have two pipelines, a merge request pipeline that creates validates, fmt's and plans and a main pipeline that fetches the plan from the last MR and applies it.

When integrating all 23 of our environments we realized that only 20 were being fetched during the fetch-mr-plan step. The rest cannot be found.

Steps to reproduce

  1. Create a pipeline that generates plans. The pipeline must include > 20 entries in the matrix, ex:
plan:
  extends: [.opentofu:plan]
  parallel: *common_matrix
  rules:
    - if: $CI_PIPELINE_SOURCE == 'merge_request_event'

NOTE: common_matrix is a list 23 entries such as:

.commin_matrix: &common_matrix
  matrix:
    - GITLAB_TOFU_ROOT_DIR: client_TLA1_stage/
      GITLAB_TOFU_STATE_NAME: client_TLA1_stage
  1. Create a pipeline that fetches plan from MR
fetch_plan:
  extends: [.opentofu:fetch-mr-plan]
  parallel: *common_matrix
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH && $CI_PIPELINE_SOURCE == "push"
  1. When running the fetch_plan, the following error will appear for the last 3 plans (since the limit is 20 and we have 23)
ERROR: unable to find job ID for 'plan: [client_TLA_stage/, client_TLA_stage]' job in pipeline with ID <JOB ID>.

Example Project

n/a

What is the current bug behavior?

API calls are limited to 20

What is the expected correct behavior?

API calls should adhere to pagination

A quick win for us would be to set the per_page param to 100 or so.

Relevant logs and/or screenshots

Please see above

Output of checks

n/a

Results of GitLab environment info

We are using version 3.7.0 of the opentofu component.o

Results of GitLab application Check

n/a

Possible fixes

In gitlab code:

https://gitlab.com/components/opentofu/-/blob/main/src/gitlab-tofu.sh?ref_type=heads#L413

resp="$(
  curl \
    --silent --show-error --fail-with-body \
    --header "PRIVATE-TOKEN: ${GITLAB_TOFU_TOKEN}" \
    "${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/pipelines/${pipeline_id}/jobs"
)"

job_id="$(echo "${resp}" | jq --arg job_name "${fetch_mr_plan_plan_job_name}" '.[] | select(.name==$job_name) | .id')"

Add pagination / per page limit to the above call and perhaps others.