Race Condition in `default_branch_protection` handling when using the API to set it on instance-level

We've got some users experiencing issues with the Terraform Provider for GitLab when creating a new project. Besides these issues (for which I may create some more issues here, since I think that there are more bugs / strange undocumented behaviors) I've found a weird behavior during my attempts to reproduce it.

When changing the default_branch_protection on instance-level, it seems not to be applied instantly. Consider the following script (an API and GITLAB_TOKEN variable with admin access is required):

#!/bin/sh

set -x

curl -X PUT "$API/application/settings" --header "PRIVATE-TOKEN: $GITLAB_TOKEN" --header "Content-Type: application/json" --data '
{
 "default_branch_protection": 0
}
'

# wait for it to propagate - whatever that means, but that's the point of this issue after all :D
sleep 10

curl -X PUT "$API/application/settings" --header "PRIVATE-TOKEN: $GITLAB_TOKEN" --header "Content-Type: application/json" --data '
{
 "default_branch_protection": 2
}
'

project=$(curl -X POST "$API/projects" --header "PRIVATE-TOKEN: $GITLAB_TOKEN" --header "Content-Type: application/json" --data '
{
 "name": "fun-with-default-branch-protections",
 "default_branch": "main",
 "initialize_with_readme": true
}
')
project_id=$(echo "$project" | jq -r '.id')
curl -X GET "$API/projects/$project_id" --header "PRIVATE-TOKEN: $GITLAB_TOKEN" | jq

... running it a few times will eventually end up in a situation where the default branch in the fun-with-default-branch-protection repository is not protected (also verified in the UI).

Now my question: what exactly happens when the default_branch_protection setting with the application/settings API endpoint is being set? Does this trigger some kind of background task to propagate it? Or is there some caching going on?

What's a reliable way to know if that has happened? Until when after setting the value in the API can I expect new projects to be created according to it?

/cc @nagyv-gitlab @nmezzopera may I once again ask for your triage wizardry ? I don't have time to dig into it myself right now ... 😿

Edited by See GitLab account @timofurrer