Retry and close master broken incidents if failed with transient error

Retry and auto-close master broken incidents (!2151 - merged) was reverted due to a flaw in the retry logic when failure isn't transient.

This MR resubmits Retry and auto-close master broken incidents (!2151 - merged), and added 1 fix:

Only retry job/pipeline if we know that all of the failures are due to transient errors.