resource_group process_mode ordered should have option to not include upcoming jobs

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

Background

Original issue and MR:

Defined the process_modes as follows:

    def upcoming_processables
      if unordered?
        processables.waiting_for_resource
      elsif oldest_first?
        processables.waiting_for_resource_or_upcoming
          .order(Arel.sql("commit_id ASC, #{sort_by_job_status}"))
      elsif newest_first?
        processables.waiting_for_resource_or_upcoming
          .order(Arel.sql("commit_id DESC, #{sort_by_job_status}"))
      else
        Ci::Processable.none
      end
    end

Problem to solve

The issue is that changing the mode from unordered to one of the ordered options also changes the selection behavior to include created which introduces a new state where nothing runs. The original issue seemed to describe the behavior as random if it does not include created. I agree that without Skip outdated deployment jobs setting enable this would be the case. The original issue seems to discuss the point of this feature being to work with that setting enabled. Perhaps there is an odd use case, but I cannot see one for newest_first without having Skip outdated deployment jobs enabled.

oldest_first makes sense if you want to make sure every job gets a chance to run and care about order. I cannot see a use case for reverse order, but only when merged rapidly?

With newest_first, if I merge after all jobs complete then I get oldest_first behavior, but if I merge rapidly then I get reverse order.

Proposal

Our desire is for deployments to be skipped when newer ones are waiting to run, but to avoid canceling running deployments and avoid concurrent deployments. Seemingly what we want is:

deploy:
  environment:
    name: production
  resource_group: production

With newest_first where it does not consider waiting_for_resource_or_upcoming jobs, but only those in the waiting_for_resource. Otherwise, if one merges rapidly (faster than the dependencies of the deploy job (like container builds)) then no jobs will run as they will always wait on the newest created job. For our project this could mean no deployment for hours during prime time.

unordered is truly random and seemingly has no benefits.

The behavior of including created seems to change the definition of the resource_group a bit beyond a lock since those jobs are not actively in contention for the lock, but are still blocking other jobs from obtaining it.

Unless there is a strong use-case newest_first it seems like it ought to be changed to waiting_for_resource instead of waiting_for_resource_or_upcoming. I could also see an argument for changing the default "unordered" to be newest_first for just waiting_for_resource. That way if folks have Skip outdated deployment jobs enabled they get the rather intuitive behavior of the newest waiting deployment running and skipping the middle ones. Otherwise, I could see adding new modes that are sorted, but only on waiting_for_resource like newest_first_pending_only.

CC @Istador @shinya.maeda

Edited by 🤖 GitLab Bot 🤖