Aync index creation improvements

Failing async index creation can block the rest of the queue, as it happened with !156090 (comment 2127967577). To handle such cased better we can:

  • When picking index from the pending queue, skip entries with failed attempts above a predefined limit.
  • Update prepare_async_index to check if the target table is partitioned, and delegate to prepare_partitioned_async_index if needed.
  • Sort by attempts DESC (see #494002 (comment 2129960177))
  • Log failures, and send alerts to #database channel. See example of watcher for Monitoring Locked Tables: gitlab-com/runbooks!6802 (merged)
Edited by Omar Qunsul