Ensure proper handling of recoverable and irrecoverable errors

Summary

Context: !503 (comment 1068371082)

@pursultani:

This goes beyond this MR, but we have a major problem with classifying errors, those that we can recover from, or hope that they will go away with retry, e.g. failed to create an object because API server timed out, and those that we can recover from and retrying is pointless Custom Resource changes, for example the shared-secrets Job fail due to a configuration error.

Acceptance criteria

• Recoverable errors are mitigated and scheduled for a requeue
• Irrecoverable errors do not schedule a requeue
• (Any other error-related optimization)