Currently, Auto DevOps runs jobs even when no runner is configured for the project. This causes the jobs to hang indefinitely providing no real value to end users.
For example, projects that are either A) sample projects never meant to be run on CI or B) projects not meant to built on GitLab CI (but another CI system instead) will both run and show hanging pipelines.
Proposal
Extend Auto DevOps logic which decides whether or not to run a pipeline to include the presence of a runner.
Do not run (or auto enable) Auto DevOps for projects that do not have a shared runner configured.
If user tries to manually enable Auto DevOps, present clear messaging of what's required for Auto DevOps.
Will still run for explicitly enabled auto devops, regardless of runner configuration
What does success look like, and how can we measure that?
(If no way to measure success, link to an issue that will implement a way to measure this)
Daniel Gruessochanged title from Don't run Auto DevOps for projects that don't hav a shared runner configured to Don't run Auto DevOps for projects that don't have a shared runner configured
changed title from Don't run Auto DevOps for projects that don't hav a shared runner configured to Don't run Auto DevOps for projects that don't have a shared runner configured
Daniel Gruessochanged title from Don't run Auto DevOps for projects that don't have a shared runner configured to Don't run Auto DevOps for projects which don't have a shared runner configured
changed title from Don't run Auto DevOps for projects that don't have a shared runner configured to Don't run Auto DevOps for projects which don't have a shared runner configured
Also this needs more consideration because are we just talking about auto enabling (ie. honouring the instance wide configuration) or are we taking this further and saying we would never run auto devops without a runner. What happens if they are removing their runner and adding another one? We would lose pipelines...
Daniel Gruessochanged title from Don't run Auto DevOps for projects which don't have a shared runner configured to Don't run Auto DevOps for projects which don't have a runner configured
changed title from Don't run Auto DevOps for projects which don't have a shared runner configured to Don't run Auto DevOps for projects which don't have a runner configured
My current understanding is that the logic is quite involved to determine which runner can pick up which pipeline. My worry is that we might have to create the pipeline first before we can currently determine if it can be picked up by a runner. The current UX for any pipeline is that a pipeline becomes "stuck" (with a reason if known) if GitLab detects that no runner can pick this pipeline.
In any case, I think that we should think about a similar case where we create a manual pipeline. Does that screen currently alert the user if no runner can pick up a pipeline from this project ?
~"CI/CD" might be able to help here too to consult with.
My worry is that we might have to create the pipeline first before we can currently determine if it can be picked up by a runner
I don't think this is true. I think we can safely do Project#any_runners?. Performance may be a concern but I think it's probably fine.
@ayufan can you think of any other potential problems with this approach? I know this deviates us a little further from a normal CI template but I think it is warranted in the case of Auto DevOps as we are implicitly running the pipeline for the user it does make some sense that we should avoid this if there is no runner configured to run the pipeline anyway.
Even more confusingly if we change the definition of Project#auto_devops_enabled? so that it's false if there is no runner then the CI/CD > Pipelines page will show empty state and the user won't even be able to run a pipeline.
@tkuah I like this idea, because it doesn't complicate the definition of when we create pipelines, and avoids cases like:
missing the creation of pipelines during a runner maintenance window
obscuring why Auto DevOps is not working when configuring GitLab for the first time
However, if my understanding of the code is correct, the combination of StuckCiJobsWorker and the auto-disabling of Auto DevOps on failure should already do this: If a build gets stuck, it gets dropped and transitions to failed, and if it was part of an Auto DevOps pipeline, it disables Auto DevOps after the failure.
Still, it doesn't solve the problem that users will get notified about the failure of something they likely did not ask for. Instead, it looks like they will get two emails: One for the pipeline failure, and another one to tell them that Auto DevOps got disabled.
Can you elaborate? Do you mean that a stuck runner would explicitly enable Auto DevOps? Or only if it was ran manually?
To give more background, currently StuckCiJobsWorker will run every so often and fails any jobs that are stuck as described above by @hfyngvason. But this means that the project will always get the failed commit status as well as two emails.
I wondered if cancelling (implicitly Auto DevOps) jobs instead of failing them will avoid some of these issues.
Still, it doesn't solve the problem that users will get notified about the failure of something they likely did not ask for. Instead, it looks like they will get two emails: One for the pipeline failure, and another one to tell them that Auto DevOps got disabled.
Perhaps the emails are the problem then? We assumed the emails would create less confusion but maybe we'd actually prefer to change our logic like:
If Auto DevOps ran for the first time and it was implicitly enabled then we silently disable it without sending a failure notification
It's kinda the opposite of what we have now but maybe creating less noise here would create less concern?
After playing around with the code and interacting with the different solutions, I'm leaning towards just going forward with the original suggestion (implicitly disabling auto devops if no runner present).
Implicitly cancelling a pipeline (or failing but not notifying) has additional UX concerns that we'd need to work out. We will have questions such as "Why do I have a stuck/cancelled/failed pipeline?". It still makes things more complicated, just in a different way.
My only concern at this point is the chance of missing a pipeline because all runners were temporarily unavailable. I don't think this is a big risk. What do you think?
I would like to be linked to the original concerns regarding this before making a decision here. Its not clear what specific problem we are solving and that will dictate which direction we go in. Do users not want to see pipelines ran in certain projects for some specific reasons? Or is it that failed pipelines block workflows for some customers? Are the notifications annoying? All the above?
@danielgruesso can you link to the conversations you have had?
@tauriedavis I went through my notes, don't have public links to share as I made these in local files mostly (something I've stopped doing since). I've updated the description with the use cases. Most of the issues users experienced will be avoided with the implementation of https://gitlab.com/gitlab-org/gitlab-ce/issues/57483, however, if we have a simple way to solve this problem I'd still like to see that through.
Because we have an acceptable workaround to the problems stated in the description (with the disablement of Auto DevOps at various levels) AND we will address most of the issues around not triggering Auto DevOps when it won't work here https://gitlab.com/gitlab-org/gitlab-ce/issues/57483 we will put this one back on the backlog for now until we evaluate how https://gitlab.com/gitlab-org/gitlab-ce/issues/57483 will impact the usage.
GitLab is moving all development for both GitLab Community Edition
and Enterprise Edition into a single codebase. The current
gitlab-ce repository will become a read-only mirror, without any
proprietary code. All development is moved to the current
gitlab-ee repository, which we will rename to just gitlab in the
coming weeks. As part of this migration, issues will be moved to the
current gitlab-ee project.
If you have any questions about all of this, please ask them in our
dedicated FAQ issue.
Using "gitlab" and "gitlab-ce" would be confusing, so we decided to
rename gitlab-ce to gitlab-foss to make the purpose of this FOSS
repository more clear
I created a merge requests for CE, and this got closed. What do I
need to do?
Everything in the ee/ directory is proprietary. Everything else is
free and open source software. If your merge request does not change
anything in the ee/ directory, the process of contributing changes
is the same as when using the gitlab-ce repository.
Will you accept merge requests on the gitlab-ce/gitlab-foss project
after it has been renamed?
No. Merge requests submitted to this project will be closed automatically.
Will I still be able to view old issues and merge requests in
gitlab-ce/gitlab-foss?
Yes.
How will this affect users of GitLab CE using Omnibus?
No changes will be necessary, as the packages built remain the same.
How will this affect users of GitLab CE that build from source?
Once the project has been renamed, you will need to change your Git
remotes to use this new URL. GitLab will take care of redirecting Git
operations so there is no hard deadline, but we recommend doing this
as soon as the projects have been renamed.
Where can I see a timeline of the remaining steps?