Validating the **contents** of the .gitlab-ci.yml file - Provide a real example that reads from a .gitlab-ci.yml that **documents** and explains the process of how it is converted from yaml to JSON

Documntation on API linting API linting could be extended to give an example of POST request to https://gitlab.com/api/v4/ci/lint with the POST request body reading the contents of a .gitlab-ci.yml file as opposed to just including a json payload that is already converted. What steps were taken to convert the YAML file to the JSON payload?

I have asked a question on how to achieve this. This documents my attempts so far with writing a bash script that uses ruby to convert the .gitlab-ci.yml file to json and send that in the request payload. Asked a question earlier, here and no responses.

Can anyone from GitLab step in and provide a working example that reads from a .gitlab-ci.yml and POSTS the content to the API for linting? Also could the API be extended to accept directly the yml contents of the file?

Hi,

Trying to make a curl request to gitlab.com api for linting .gitlab-ci.yaml file but receiving bad request response: {"status":400,"error":"Bad Request"}

#!/usr/bin/env bash


PAYLOAD=$( cat << JSON 
{ "content":
$(<$PWD/../.gitlab-ci.yml)
JSON
)

echo "Payload is $PAYLOAD"

curl --include --show-error --request POST --header "Content-Type: application/json" --header "Accept: application/json" "https://gitlab.com/api/v4/ci/lint" --data-binary "$PAYLOAD"

Has anyone managed to successfully lint a .gitlab-ci.yml via a bash script?

Update

I think what is happening is that the GitLab CI endpoint expects the contents of the .gitlab-ci yaml file to be converted to json for the POST request, as opposed to directly sending the contents of the yaml file. See here.

So...I modifed the script to use ruby to convert yaml to json before sending and this works for very simple .gitlab-ci.yml.

#!/usr/bin/env bash

json=$(ruby -ryaml -rjson -e 'puts JSON.pretty_generate(YAML.load(ARGF))' < .gitlab-ci.yml)

# escape quotes
json_content=$(echo $json | perl -pe 's/(?<!\\)"/\\"/g')


# Add object contect for GitLab linter
json_content='{"content": "'${json_content}'"}'

echo "${json_content}"

curl --include --show-error --request POST \
    --header "Content-Type: application/json" \
    --header "Accept: application/json" \
    "https://gitlab.com/api/v4/ci/lint" \
    --data-binary "$json_content"

However when using the yaml file for my project it gives an error: {"status":"invalid","errors":["(\u003cunknown\u003e): did not find expected ',' or ']' while parsing a flow sequence at line 1 column 221"]}% When I use the gitlab web page for linting the file is valid.

{"content": "{ \"stages\": [ \"build\", \"test\", \"pages\", \"release\" ], \"variables\": { \"DOCKER_DRIVER\": \"overlay2\" }, \"services\": [ \"docker:19.03.11-dind\" ], \"build:plugin\": { \"image\": \"docker:19.03.11\", \"stage\": \"build\", \"before_script\": [ \"echo \"$CI_JOB_TOKEN\" | docker login -u gitlab-ci-token --password-stdin \"$CI_REGISTRY\"\" ].....

Column 221 is \"image\": \"docker:19.03.11\" in the above json extract, specifically at the closing escaped quote. Think it is a problem with incorrectly escaped quotes??

Also tried with other complex files and get the same error. Very confused on how to use the [api]((https://docs.gitlab.com/ee/api/lint.html) to correctly validate gitlab-ci.yml from file.....Can anyone from GitLab help with this??

Second Update

Wrote this bash script:

#!/usr/bin/env bash

json=$(ruby -ryaml -rjson -e 'puts JSON.pretty_generate(YAML.load(ARGF))' < custom_hooks/valid.yml)

# escape quotes
json_content=$(echo $json | perl -pe 's/(?<!\\)"/\\"/g')


# Add object contect for GitLab linter
json_content='{"content": "'${json_content}'"}'

# Output escaped content to file
echo "$json_content" > custom_hooks/input.json
echo "Escaped json content written to file input.json"

curl --include --show-error --request POST \
    --header "Content-Type: application/json" \
    --header "Accept: application/json" \
    "https://gitlab.com/api/v4/ci/lint" \
    --data-binary "$json_content"

Using the above bash script.... this yaml file:

stages:
  - test
test:
  stage: test
  script:
    - echo "test"

gets converted to this json:

{"content": "{ \"stages\": [ \"test\" ], \"test\": { \"stage\": \"test\", \"script\": [ \"echo \"test\"\" ] } }"}

When this is sent to the api the following json error response is received:

{"status":"invalid","errors":["(\u003cunknown\u003e): did not find expected ',' or ']' while parsing a flow sequence at line 1 column 62"]}%

Can anyone please provide a clear documented example how to use GitLab's api correctly, starting from reading the contents of a .gitlab-ci.yml file?

Finally Managed To Use The API to validate YAML file

Got it working finally using the following script:

#!/usr/bin/env bash

json=$(ruby -ryaml -rjson -e 'puts(YAML.load(ARGF.read).to_json)' custom_hooks/valid.yml)

# escape quotes
json_content=$(echo $json | python -c 'import json,sys; print(json.dumps(sys.stdin.read()))')
echo $json_content

# Add object contect for GitLab linter
json_content="{\"content\": ${json_content}}"

curl --include --show-error --request POST \
    --header "Content-Type: application/json" \
    --header "Accept: application/json" \
    "https://gitlab.com/api/v4/ci/lint" \
    --data-binary "$json_content"

N.B will be tweaking script to read file from system args rather than the fixed file location custom_hooks/valid.yml. Including this script on the offchance that it will help others.

The problem was that initially I was sending YAML contents of the file directly to the api:

{ "content": { <contents of .gitlab-yml> } }

It looks as though GitLab accepts YAML converted to an escaped JSON string in their API. So used ruby to convert the yaml to JSON and then used python to escape the resulting JSON produced by ruby. Finally was able to use curl to send the escaped JSON string to the GitLab API for validating.....

Not sure if Ruby has something equivalent to python's json.dumps .... but this solution allows me to validate gitlab-ci....Next stage hookup to git pre-commit hooks / server side pre-receive (if possible!) to prevent invalid .gitlab-ci.yml files breaking CI pipeline.

Edited by dcs3spp