Follow-up from "Disallow retry of old builds": Improve UX experience

The following discussion from !22538 (merged) should be addressed:

Edited by Kamil Trzciński