Repository daily maintenance fails to complete because of DeadlineExceeded error
Gitaly has a background job that runs by schedule each day. The task of the job is to run OptimizeRepository
RPC on a randomly chosen repository. The task runs for a certain period of time that is configured via daily_maintenance.duration
setting. And a usual context.WithTimeout(ctx, schedule.Duration.Duration())
is used to cancel execution context and interrupt execution of the task. Because of that in-flight RPC call is cancelled as well, so it is not a graceful termination. This leads to the rpc error: code = DeadlineExceeded desc = could not repack: rpc error: code = Internal desc = signal: terminated
error as the result of the /gitaly.RepositoryService/OptimizeRepository
RPC call. And on the calling side the call finishes with maintenance: repo optimization failure
, but actually it is not a real error as we should stop job gracefully.
/cc @timzallmann