Allow CI jobs to decide whether they are "allowed to fail" programmatically

Currently CI jobs can be set as "allowed to fail" in which case they show up as an exclamation mark in the pipeline, but the pipeline can continue as though they passed. This is useful to allow "warning" states to appear in the pipeline:

failed job	failed but "allowed to fail"

However, this is not quite as flexible as it could be. I think it could be extremely useful to allow a job to be "allowed to fail" based on criteria from the job itself.

We may have a linter which produces both warnings and errors. Perhaps we want it to be "allowed to fail" if there are warnings, but do a hard fail if there are errors. Or perhaps our "danger" job could fail the pipeline based on some criteria, but just produce a warning based on others.

In gitlab-org/gitlab-ce!21859 we were contemplating a merge of the static-analysis and danger-review jobs into a single job since they have overlapping concerns and this is one major hangup for that.