Serialization Error with External Status Check

Summary

It appears that when using the External Status Check to fire off a CI Pipeline via a Trigger Token, the serialized JSON payload that is sent within the TRIGGER_PAYLOAD is improperly serialized which causes issues when unpacked.

{"{\"object_kind\":\"merge_request\",\"event_type\":\"merge_request\",\"user\":{\"id\":428430,\"name\":\"Tim Poffenbarger\",\"username\":\"poffey21\",\"avatar_url\":\"https://gitlab.com/uploads/-/system/user/avatar/428430/avatar.png\",\"email\":\"":{"REDACTED":{"\"},\"project\":{\"id\":34913679,\"name\":\"Django Auto DevOps\",\"description\":\"":{"Sprint 2":{"(/groups/tech-marketing/demos/gitlab-agile-demo/large-co/-/issues?scope":"all"}}}},"utf8":"✓","state":"opened","milestone_title":"CY20 Q2 Sprint 2) | [Value Stream](/groups/gitlab-org/-/analytics/value_stream_analytics)\",\"homepage\":\"https://gitlab.com/gitlab-gold/tpoffenbarger/django\"},\"assignees\":[{\"id\":5056786,\"name\":\"Joe Randazzo\",\"username\":\"jrandazzo\",\"avatar_url\":\"https://secure.gravatar.com/avatar/dbbcc6722f753b2acad3436217cda749?s=80","d":"identicon\",\"email\":\"[REDACTED]\"}],\"external_approval_rule\":{\"id\":415,\"name\":\"Requirements Management\",\"external_url\":\"https://gitlab.com/api/v4/projects/38790749/ref/main/trigger/pipeline?token=01bc0798bb1097093f38ba139e5fc9\"}}","id":"38790749","ref":"main","variables":{}}

👆🏼 specifically problems arise at the first mention of redacted email \"email\":\"":{"REDACTED":{"\"} (there are two email references).

Steps to reproduce

The following steps will most likely need to be performed by a non-administrator and specifically on SaaS as the "Redacted" might be specific to SaaS.

  1. Create a simple project like this to output the TRIGGER_PAYLOAD data when the CI runs.
  2. Generate a Trigger Token in a project: Project > Settings > CI/CD > Pipeline triggers > Add trigger (trigger tokens will, unlike any other token to trigger a pipeline, expose the Trigger Payload Data to project owners within the GitLab UI when viewing the job).
  3. Create a Webhook and pass the generated trigger from step 1: Project > Settings > Webhooks where URL is https://gitlab.com/api/v4/projects/PROJECT_ID/ref/REF_NAME/trigger/pipeline?token=TOKEN (replacing the all-caps values with appropriate variables). Example: https://staging.gitlab.com/api/v4/projects/7947124/ref/main/trigger/pipeline?token=glptt-b5e28a98cf4bc54f3331985c160d6a0fdb1e6791
  4. Test the webhook and click on "Merge requests events" image
  5. Check the recent events of that webhook and see that the payload is a valid JSON payload.
  6. Now add an external status check for this project by going to Project Settings > General > Merge requests.
  7. Add status check and put the same trigger URL as the URL in "API to check".
  8. Create an MR in this project so that the external status check we created above is triggered.
  9. Check the output of the pipeline that was triggered because of the external status check. The TRIGGER_PAYLOAD data is not correctly escaped and is not a valid JSON payload. Example job - https://staging.gitlab.com/hiftikhar/issue-371609/-/jobs/43509743
Valid JSON payload Invalid JSON payload
image image

Example Project

Python Project - https://gitlab.com/gitlab-gold/tpoffenbarger/dart/-/jobs/2917587532#L158

Ruby Project - https://staging.gitlab.com/hiftikhar/issue-371609

Please note that the Trigger Token that is mentioned in the description has been revoked and is no longer valid.

What is the current bug behavior?

Invalid JSON is present in the TRIGGER_PAYLOAD.

What is the expected correct behavior?

JSON should be able to be both serialized and deserialized.

Relevant logs and/or screenshots

Output of checks

Results of GitLab environment info

Expand for output related to GitLab environment info 

(For installations with omnibus-gitlab package run and paste the output of:
`sudo gitlab-rake gitlab:env:info`)

(For installations from source run and paste the output of:
`sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production`)

Results of GitLab application Check

Expand for output related to the GitLab application check 
(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:check SANITIZE=true)


(For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true)


(we will only investigate if the tests are passing)

Possible fixes

Add headers: { 'Content-Type': 'application/json' } while sending the POST request to the external status check URL.

Edited by Huzaifa Iftikhar