Fix: Tf state file not deletable with $CI_JOB_TOKEN

• Please check this box if this contribution uses AI-generated content (including content generated by GitLab Duo features) as outlined in the GitLab DCO & CLA

What does this MR do and why?

This MR allows maintainers to use $CI_JOB_TOKEN to delete Terraform state file by updating the DELETE endpoint to accept both request and basic_auth as authentication methods. I followed discussion in #375242 to guide my implementation.

MR acceptance checklist

Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Screenshots or screen recordings

Screenshots are required for UI changes, and strongly recommended for all other merge requests.

Before	After

How to set up and validate locally

1. Create a project with CI config in GDK

include:
  template: Terraform.latest.gitlab-ci.yml

delete:
  image: curlimages/curl
  script:
    # Delete the Terraform state
    - 'curl --header "Job-Token: ${CI_JOB_TOKEN}" --request DELETE "${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/terraform/state/default"'
    
    # Verify deletion (should return 404)
    - |
      response=$(curl -s -o /dev/null -w "%{http_code}" --header "Job-Token: ${CI_JOB_TOKEN}" "${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/terraform/state/default")
      if [ "$response" = "404" ]; then
        echo "Terraform state successfully deleted"
      else
        echo "Unexpected response: $response"
        exit 1
      fi