Resolve "Race condition in fetching Kubernetes token causes missing `$KUBECONFIG`"

What does this MR do?

Retry fetching Kubernetes Secret#token (#63507 (closed))

Since Kubernetes is creating the Secret and token asynchronously it is necessary that we implement some delay or retrying logic to avoid a race condition where we fetch a Secret before the token is even set. There does not appear to be any way for us to force it to be set with any synchronous API call so retrying seems to be the only option.

NOTE: Unfortunately it was not possible to simply retry the worker responsible for this as every time the worker kicks off it restarts the async process. As we learnt testing with a customer this is pathological and you just need some delay between touching the service account and fetching the token.

Does this MR meet the acceptance criteria?

Conformity

• Changelog entry
• [-] Documentation created/updated or follow-up review issue created
• Code review guidelines
• Merge request performance guidelines
• Style guides
• Database guides
• Separation of EE specific content

Performance and testing

• Review and add/update tests for this feature/bug. Consider all test levels. See the Test Planning Process.
• Tested in all supported browsers

Security

If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:

• Label as security and @ mention @gitlab-com/gl-security/appsec
• The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
• Security reports checked/validated by a reviewer from the AppSec team

Closes #63507 (closed)