Startup probe failure since June 12th
Problem to solve
Continuous deployments via Runway are blocked by startup probe failure. https://gitlab.com/gitlab-org/modelops/applied-ml/code-suggestions/ai-assist/-/pipelines?page=1&scope=all&ref=main
e.g.
RCA
Confirmed that d28e279c is causing this failure.
The async init for resources implicitly enable the Async mode in the upstream providers. This effectively converts the result of dependency resolution to awaitable. Hence, errors like AttributeError: '_asyncio.Future' object has no attribute 'execute'"
are observed.
See https://python-dependency-injector.ets-labs.org/providers/async.html.
A quick fix would be to put await
for all of the affected providers.
Corrective action
Surprisingly, we couldn't catch this error in MR pipelines.
This type of error should be able to be caught by integration tests. Currently, @alipniagov is working on it in chore: Add e2e test for CS user auth flow (!919 - merged), which would provide sufficient coverage to avoid this error in the future.