Skip to content

Implement startup probe for MG

Dylan Bernardi requested to merge add-startup-probe into main

🧩 Problem to solve

Per this issue, there is no startup probe for the codegen model.

💡 Proposal

This probe should verify if the codegen model is ready. The required grpc endpoint is available through the InferenceServerClient.

️ Plan of attack

This implementation adds a startup probe to the model-gateway yaml.

Notes:

Might want to edit the timing on startup and liveness probes. As per this implementation, it appears that the timing is different in this implementation in SR (here)

@dbernardi to investigate this further, but can be fixed in a merge request later if need be.

cc @mray2020 @AndrasHerczeg

Edited by Dylan Bernardi

Merge request reports