Deployment checks are causing for the deployment jobs to fail

On #1241 (closed) we enabled TRIGGER_INTERMEDIATE_PRODUCTION_CHECKS on deployer and a FF https://ops.gitlab.net/gitlab-org/release/tools/-/feature_flags/185/edit on release-tools, which allows keeping track of an on-going deployment by sending notifications to #f_upcoming_slack channel.

This suddenly caused gprd-migrations to failed:

fatal: [deploy-01-sv-gprd.c.gitlab-production.internal]: FAILED! => 
  msg: 'template error while templating string: expected token ''end of print statement'', got ''string''. String: {{ lookup(''env'', ''IGNORE_PRODUCTION_CHECKS'') | default ''false'' }}'

The env variable was removed from deployer, the job was restarted and it succeeded this time, which makes me think it's related to TRIGGER_INTERMEDIATE_PRODUCTION_CHECKS. From Slack - Internal link only:

rspeicher: mayra https://ops.gitlab.net/gitlab-com/gl-infra/deployer/-/jobs/2257952 failed here. msg: 'template error while templating string: expected token ''end of print statement'', got ''string''. String: {{ lookup(''env'', ''IGNORE_PRODUCTION_CHECKS'') | default ''false'' }}'

mayra: oh well deactivating the feature 😅

mayra: Done, should we retry?

skarbek: i find it difficult to believe this would be related....

skarbek: i'll hit the button for retry

rspeicher: Why is it showing '' instead of "?

mayra: Why do you think it’s not related skarbek? 🤔

skarbek: ... it succeeded

skarbek: i think unrelated because this was only looking up an environment variable to populate a string variable inside of ansible

skarbek: as if the lookup function in python had failed in my opinion