When Auto DevOps is enabled, evaluate if a build mechanism exists before running. If no build mechanism, disable Auto DevOps. Mechanisms to consider should include:
Presence of a Dockerfile
Presence of matching file for buildpack bin/detect (ie Gemfile for Ruby, package.json for node.js, etc). We'll just need to hardcode all the files listed in bin/detect for all the supported buildpacks.
Presence of a custom BUILDPACK_URL environment variable
This should only happen when Auto DevOps is "implicitly" enabled and not when explicitly enabled as someone that has explicitly enabled Auto DevOps will want to see a failing pipeline to understand why it's not working.
We should also ensure the logic to check if the files exist is based on the SHA for which we are currently creating a pipeline (and not just looking at master for example).
Lastly, if the check runs and no Dockerfile or matching buildpack are found AND Auto DevOps has been enabled implicitly for the project in question, we should inform user about this fact in the Auto DevOps settings section by displaying a warning alert that reads:
You must add a Dockerfile or supported buildpack in order for the Auto DevOps pipeline to run. More information
If the correct file is added, we should automatically run the pipeline.
What does success look like, and how can we measure that?
We should track the instances where this evaluation returned true or false to start.
Do we have a related issue for where AutoDevOps is possible (i.e., there is code or a website or whatever to build) but the person is using a different CI/CD platform and doesn't want this to be running for their project?
Daniel Gruessochanged title from Don't run Auto DevOps for projects w/o build mechanism to Don't run Auto DevOps when no dockerfile or matching buildpack exists
changed title from Don't run Auto DevOps for projects w/o build mechanism to Don't run Auto DevOps when no dockerfile or matching buildpack exists
Do we have a related issue for where AutoDevOps is possible (i.e., there is code or a website or whatever to build) but the person is using a different CI/CD platform and doesn't want this to be running for their project?
@jlenny not that I can remember. Feel free to create one but we'd need more information around how we could detect that they were using a different CI/CD system.
@tauriedavis one UX consideration I had here is what we would show in our UI for projects that do not match Auto DevOps.
Right now I can think of several places in the app where Auto DevOps being enabled is visible to the user -
the settings where it is enabled implicitly or explicitly
the banner which shows up on the project
I'd be curious if these things should still appear the same to the user if their project's master branch (needed to get technical there considering we need to check the files from a specific branch) does not have any of the files to indicate that it is compatible.
I'm thinking it would be strange if we say "your project will run Auto DevOps without any .gitlab-ci.yml" in a banner but then we actually won't. But then the curious thing would be that if you added a Gemfile or something else we will actually kick off the pipeline.
Also there are a bunch of weird edge cases I can think of here that make my head spin. Like for example if the user explicitly did enable Auto DevOps then we probably do want to kick off the pipeline for them regardless of the files because they will be confused why nothing is happening. But the way our UI works there is actually no way to enable Auto DevOps for projects where Auto DevOps is already implicitly enabled and as such a user that does not have a certain file in their repo will never really be able to understand why there Auto DevOps pipelines are not running.
It's possible we will want the behaviour to differ depending on if Auto DevOps was enabled explicitly or implicitly (ie. instance level) but then we the new group level implicit setting I'm kinda on the fence. Maybe they do want Auto DevOps to run for every project regardless of what files there are.
I'm thinking it would be strange if we say "your project will run Auto DevOps without any .gitlab-ci.yml" in a banner but then we actually won't.
Settings
Current copy:
Default to Auto DevOps pipeline (instance enabled)
The Auto DevOps pipeline will run if no alternative CI configuration file is found. More information
Proposed:
Default to Auto DevOps pipeline (instance enabled)
The Auto DevOps pipeline will be used if no alternative CI configuration file is found. More information
Alert
I think this copy works and doesn't need edited:
The Auto DevOps pipeline has been enabled and will be used if no alternative CI configuration file is found. More information
These statements are true. If Auto DevOps is enabled implicitly, we won't run the pipeline automatically. However, if the user manually triggers a pipeline, then the ADO pipeline will be used.
Like for example if the user explicitly did enable Auto DevOps then we probably do want to kick off the pipeline for them regardless of the files because they will be confused why nothing is happening.
Agreed
But the way our UI works there is actually no way to enable Auto DevOps for projects where Auto DevOps is already implicitly enabled
This isn't technically true. Checking and unchecking the setting makes ADO explicitly enabled. Although, I wouldn't expect a user to do this. I would think they would trigger the pipeline manually instead. Perhaps if this happens, we mark ADO as explicitly enabled? Can you think of any downsides?
but then we the new group level implicit setting I'm kinda on the fence.
This is a good point. I am also on the fence. But I imagine if a user has turned on ADO at the group level, they want it to run for all the projects in that group. And if there is a project they don't want it to run for, they would turn it off explicitly for that project. So for this issue, I think we should just start with just projects that were implicitly turned on by the instance level.
@tauriedavis I think I'm in agreement with all of your suggestions. The only thing that concerns me is separating "implicitly enabled at instance level" and "implicitly enabled at group level" might create a real mess for us. These things are closely linked in behaviour and in our code right now.
Maybe we cancel the job, rather than fail it for implicitly run ADO piplines. Would that possibly work in this scenario too?
Seems like a good idea to me too. I just don't know exactly what that will mean? Does the pipeline disappear for the user? Is it in some slightly less scary state after this? Will they still get an email saying it failed and has now been disabled for them?
I think the pipeline will still be there, but no email is sent. Don't know if users will get a scary looking commit status icon on their project homepage.
I do think it is less scary to see a grey canceled icon than a red failed icon.
Whether this addresses the problem goes back to my comment in the other runner issue
Its not clear what specific problem we are solving and that will dictate which direction we go in. Do users not want to see pipelines ran in certain projects for some specific reasons? Or is it that failed pipelines block workflows for some customers? Are the notifications annoying? All the above?
If it is a manner of failed pipelines breaking work flows and annoying notifications, then canceling the job rather than failing it would solve for this issue.
Maybe we cancel the job, rather than fail it for implicitly run ADO piplines. Would that possibly work in this scenario too?
The main problem is failures in a project as a false alarm as well as blocking changes from being merged due to a pipeline that was not initiated by the user. The cancelled pipeline would address both BUT it will still create confusion and questions. But it sounds like a good MVC nonetheless. It still feels the right thing to do is not run it at all, this would be the best experience. Are there any concerns with the original approach of adding this logic to the rails backend @DylanGriffith@tkuah ?
Are there any concerns with the original approach of adding this logic to the rails backend
My main concern is just that the changing when a CI job is triggered and when it is not is a far reaching thing which underpins a lot of GitLab CI and tonnes of other things visible in the UI. It's already very complicated to explain when it runs:
If you have a .gitlab-ci.yml OR if you have enabled Auto DevOps OR if you haven't disabled Auto DevOps for your project but have enabled it for the group OR if you haven't disabled it for the group but have enabled it for the instance
This rule above controls (at least the following things I know of and I'm sure I've missed some):
whether a pipeline triggers on push
banner messages
whether or not the pipelines page displays an empty state or the "Run pipeline" but is visible
And another "UNLESS" to the end of this complicates the user experience quite a bit so it concerns me for sure.
The ideal objective of Auto DevOps would be that it's "Just another CI pipeline" with no special behaviour. This makes it much easier to maintain, for users to understand and for users to configure.
I'm not sure we've landed on a solution here. There are two options:
If the project has no dockerfile or matching build pack, we detect this and do not run Auto DevOps for a project if ADO is explicitly enabled.
Pros: User is not concerned with a pipeline they didn't want.
If the project has no dockerfile or matching build pack, we cancel the job rather than fail it.
Pros: No changes to how pipelines currently run which means users who are expecting it to just run won't be confused. We can inform users why the job was canceled in the logs.
I think I am leaning towards the section option, especially if we are able to write in the job log why the pipeline was canceled. This would mean no errors/failures for users who were not expecting the pipeline to run, but also clarity for users who were.
If we went with the second option, its not clear to me when the job is canceled? At what point does that occur in the pipeline?
I thought the proposed solution was to only check the files if auto-devops would have otherwise been implicitly enabled. I think if the user has explicitly turned on auto-devops then they would probably understand what it is that is failing (they turned on a thing and it didn't work).
It could be that we display a warning on the project-level auto-devops enabled check if none of the required files are detected?
If the project has no dockerfile or matching build pack, we cancel the job rather than fail it.
I think this approach was mostly discussed in the context of not having a runner. Because a job without a runner will wait a long time and can therefore be cancelled. In the case of Auto DevOps build not matching the job will just kick off straight away and try to run the build process and hence will fail. I don't know if there's a way we can easily convert a failed pipeline into a "cancelled" pipeline.
We discussed canceling the pipeline for this issue in this thread a couple comments up. I just want to make sure you now don’t believe that is a good idea? Getting a little confused here..
@tauriedavis I personally believe no pipeline at all is better than a cancelled pipeline; the cancelled pipeline may still generate confusion and questions (just like the failed pipeline does currently).
Often our customers can be confused when they see failed Auto DevOps pipelines even though they are not using any CI for their projects at all. Here is the good example where enabled Auto DevOps caused a lot of confusion: https://gitlab.zendesk.com/agent/tickets/116484 (internal)
@danielgruesso if we had this feature https://gitlab.com/gitlab-org/gitlab-ce/issues/51159 it would allow us to basically replicate what bin/detect is doing for each buildpack. We'd just keep the list of all the files that the standard supported buildpacks are looking for in our template so we'll know that if none of those files are present we don't need to do anything:
only:-file:Gemfile-file:package.json...
I fear this may still require yet another feature of GitLab CI to make this work properly though because only clauses in CI YML don't compose very well right now (they only allow for OR between all the conditions).
This may not compose properly (logically) with the existing only clauses that we have in Auto DevOps today but actually it appears that we already are working on a flexible solution in https://gitlab.com/gitlab-org/gitlab-ce/issues/60085 to address just this problem!
Oh and one more detail is that we'll only enumerate the files expected by the supported herokuish buildpacks but of course we'll always want to run the pipeline if the user sets BUILDPACK_URL environment variable (referencing a custom buildpack). But for that we can easily just compose with our existing feature only:variables.
A detail of this plan that now concerns me is that it would require us adding the job filtering logic to every single job in Auto DevOps. Which includes all the partial templates for security jobs. This could potentially be problematic outside of Auto DevOps as it may mean that people's security jobs stop running. We really only want to apply this logic in the context of Auto DevOps since basically the rules we plan to implement are duplicating the rules of herokuish and therefore really only relate to projects built by auto build.
Further considering this I also am beginning to think that maybe these filtering rules would have only made sense in the case of implicitly enabled Auto DevOps as the explicitly enabled Auto DevOps users may be confused why no pipelines are being created at all for their project. If they explicitly enable Auto DevOps we may wish to have failed pipelines so they at least get feedback on why their project is not compatible with Auto DevOps.
Thinking this through more I think a better approach to this one may be a more direct approach to the actual problem. Let's not introduce this as new syntax to Auto DevOps but instead we just implement this logic on our backend.
New Plan
Basically in the case of implicitly enabled Auto DevOps we should add a bunch of checks before determining if we actually want to create the pipeline:
Check the project actually contains at least one of the files needed to be compatible with Auto Build (Dockerfile, Gemfile, ...) OR there is a BUILDPACK_URL environment variable set.
This 2nd rule described above can be what this issue is about.
Since this logic will sit next to the logic implemented in https://gitlab.com/gitlab-org/gitlab-ce/issues/49081 we should probably not work on those in parallel. Can keep them in the same issue I suppose but 2 developers should not work on them in parallel as it will cause conflicts.
maybe these filtering rules would have only made sense in the case of implicitly enabled Auto DevOps as the explicitly enabled Auto DevOps users may be confused why no pipelines are being created at all for their project. If they explicitly enable Auto DevOps we may wish to have failed pipelines so they at least get feedback on why their project is not compatible with Auto DevOps.
Though I wonder if some users who implicitly enable Auto Devops for their instance / group may also want some feedback about why their project is not compatible. Perhaps we can have some visual feedback somewhere in the Settings page for each project as well ?
This 2nd rule described above can be what this issue is about.
@DylanGriffith I think it's always been what this issue is about, just the "how" was not surfaced as clearly. So the change here is that the logic will not be executed by CI but rather by the rails backend, correct?
@danielgruesso I added the following note to the description:
This should only happen when Auto DevOps is "implicitly" enabled and not when explicitly enabled as someone that has explicitly enabled Auto DevOps will want to see a failing pipeline to understand why it's not working.
@DylanGriffith@danielgruesso I believe we discussed not disabling Auto DevOps, but simply not running the pipeline, correct? That way, if one is added then ADO will run.
Yeah from my perspective I think it doesn't make sense to disable Auto DevOps. The user won't really get any benefit from that. Leaving it enabled (albeit only implicitly enabled because we're only talking about when it's implicitly enabled for now) will mean that if they did add a matching file later they can get the same behaviour where they see a pipeline start to run for them. Assuming this behaviour was going to delight users to begin with I think it makes sense to keep it enabled.
In our team call [recording], we decided that we would not run the pipeline if no Dockerfile or matching buildpack are found and ADO has been enabled implicitly. We will also inform the user of how to get ADO to run in their settings:
You must add a Dockerfile or supported buildpack in order for the Auto DevOps pipeline to run. More information
If the user adds the necessary file, then the ADO pipeline should run automatically.
Quick update on this. The logic for this feature is pretty simple. Check if any of a list of files exist in the repo at whatever ref CI is running on (probably master, but not for MRs?).
My approach so far, so that I don't miss any cases, was to purposely add a new required argument (the repo ref) and see what breaks so I can see all the cases we actually use this logic but I hadn't anticipated the knock on effects of introducing repository checking code to our "implicitly enabled" auto-devops check. It's really time consuming to check these as the best indicator is CI, but there are so many failures that it takes a long time to dig through them all.
Since this hasn't even been through an initial review I think it's likely this issue wont make %12.1.
When Auto DevOps is enabled, evaluate if a build mechanism exists before running. If no build mechanism, disable Auto DevOps...We should also ensure the logic to check if the files exist is based on the SHA for which we are currently creating a pipeline
Would this be confusing as a developer if some branches don't trigger any pipelines (no matching buildpack files) but some do (have buildpack files) ?
Lastly, if the check runs and no Dockerfile or matching buildpack are found AND Auto DevOps has been enabled implicitly for the project in question, we should inform user about this fact in the Auto DevOps settings section by displaying a warning alert that reads:
I guess this is a mitigation but this check will only apply to the master branch ?
The main problem is failures in a project as a false alarm as well as blocking changes from being merged due to a pipeline that was not initiated by the user
Since we're seemingly unable to stop creating the pipelines in a performant way we're now considering a different approach here of cancelling the pipeline immediately after creating it. Leaving it in a cancelled state rather than a failed state. But I'm now concerned that this may not solve either of the problems you described here.
Do you know how we reproduce the situation where a user is blocked from merging an MR so that we can test what happens if the pipeline is cancelled?
Since we're seemingly unable to stop creating the pipelines in a performant way we're now considering a different approach here of cancelling the pipeline immediately after creating it.
Just confirming this.
I thought originally that we could check these files fast enough, but hadn't realised that the gitaly call was for the root directory only. Adding all the directory checks makes it much slower.
I think it's likely we will need many more UI tweaks to make this a good experience.
I agree, I think there may be tweaks we can make to make it more clear what exactly is happening. I think it would be fine to break them off in separate issues and start simply with canceling the pipeline. Potentially an easy improvement is to provide a reason on the pipeline and build views when it's automatically canceled.
Since we're seemingly unable to stop creating the pipelines in a performant way
I am curious about this. It feels like it should be possible to do this in a performant way. What do we gain by cancelling the pipeline instead of performing a background job? Is it not possible to do this with a worker?
It feels like it should be possible to do this in a performant way.
Yeah. It may be that there's more work to be done here. The way we get the file listings at the moment is effectively git ls-files as an RPC call to gitaly. Because we want to match at least as many files as herokuish we're pretty tied to their patterns, some of which are very vague and require checking every file in the repo.
What do we gain by cancelling the pipeline instead of performing a background job?
The implementation is a background job, these prerequisites run as part of pipeline creation. I don't think there's an obvious place to put this background job after a commit is pushed but before a pipeline is created. It could be that we create a new status for this instead of using cancelled. This might reflect better in the UI?
Edit: I read your question wrong. We run a background job because it makes it much easier to update the pipeline in a way that we could give the user a better experience than a failed pipeline
Is it not possible to do this with a worker?
In some ways I think this would be ideal. We could just run herokuish for real and if it fails with no matching buildpack then we somehow tell the pipeline what happened so we can show a nicer UI than a failed build. I think we decided pretty early on that this isn't trivial. I'm not sure how to go about this one.
I don't think there's an obvious place to put this background job after a commit is pushed but before a pipeline is created.
Understood, thanks. If pipeline creation happens synchronously in a web request, and making that async is off the table, there's not much we can do about that. git ls-files is pretty fast, but we must assume there be pathological repos with a very large amount of files and occasionally massive diffs.
We could just run herokuish for real and if it fails with no matching buildpack
I think your current approach is still likely a cost saver compared to running a Herokuish build, because the build would just do the same work with the added cost of the downloading the image, cloning the repo, spinning up a runner etc. Due to the runner startup costs, I think 30s in a Sidekiq background job buys us a lot more computation than 30s in a pipeline.
If preventing pipeline creation is not possible, then I agree with @tauriedavis: Start by cancelling the pipeline and iterate from there.
@DylanGriffith the specific use case came from a customer that had the Only allow merge requests to be merged if the pipeline succeeds enabled for all projects in their instance. The particular repository contained no code and changes to master were blocked when auto devops was automatically enabled, ran, and resulted in a failed pipeline.
Cancelling the pipeline is not enough for the "Pipelines must succeed" feature.
The check does allow skipped pipelines though. So the question is, do we skip the whole pipeline or do we introduce some new CI state to represent this?
Skipping works but it occurs to me that this might mean making a lot of UI conditional depending on if auto-devops was implicitly enabled or not. This MR for example would additionally need to explain why it was skipped:
This makes me thing perhaps we need some new status?
I think this is only page we would need to add an explanation of why.
This pipeline was skipped because this project is not compatible with Auto DevOps. To run Auto DevOps pipelines, ensure your project has a Dockerfile or matching buildpack.
@danielgruesso I've gone with Taurie's mock. My concern is just that we display the pipeline status in quite a few places where you might not have context of why it was skipped/canceled but perhaps it isn't a big deal if when they visit the pipeline itself, there's a message explaining it.
What if we were to delete the pipeline? Users can already do this (though admittedly only via API), and it is probably quite a common approach to get rid of an unwanted failed pipeline.
@tigerwnz I think we'll still need to store the result of the check somewhere, presumably for every commit to cover MRs/branches, so we can explain what happened or didn't happen to the user. Especially since when auto-devops is implicitly enabled - the auto-devops box is checked in settings.
I guess I could think of an alternative flow similar to how the first failing auto-devops build turns off auto-devops - we switch that to this new buildable check. Check if buildable and if not turn off auto-devops and delete the pipeline. Personally I find the existing feature too confusing. Especially since it triggers on a failed deploy which I don't think ever works first time since you need to have all the CI variables in place. Using the buildable check might improve this situation. This will only help new projects though.
I could see some value in having our own table to store the results of this check too. Especially if it could absorb the functionality of repository_languages. I.E. your project has these language splits and we think it's a rails project - have a look at our rails getting started guide...
Check if buildable and if not turn off auto-devops and delete the pipeline
@proglottis Yeah this is what I had in mind, but you're right that it isn't very obvious what has happened (and we want to raise awareness about Auto DevOps, this would be doing the opposite).
Either way I think storing this data somewhere else is a great idea. Something for a future iteration perhaps?
To better fit with CI architecture it has been suggested we implement this as a rule/match in gitlab-ci.yml. This means to implement this issue we'll first need to do https://gitlab.com/gitlab-org/gitlab-ce/issues/51159 taking into account performance as per the thread above.
GitLab is moving all development for both GitLab Community Edition
and Enterprise Edition into a single codebase. The current
gitlab-ce repository will become a read-only mirror, without any
proprietary code. All development is moved to the current
gitlab-ee repository, which we will rename to just gitlab in the
coming weeks. As part of this migration, issues will be moved to the
current gitlab-ee project.
If you have any questions about all of this, please ask them in our
dedicated FAQ issue.
Using "gitlab" and "gitlab-ce" would be confusing, so we decided to
rename gitlab-ce to gitlab-foss to make the purpose of this FOSS
repository more clear
I created a merge requests for CE, and this got closed. What do I
need to do?
Everything in the ee/ directory is proprietary. Everything else is
free and open source software. If your merge request does not change
anything in the ee/ directory, the process of contributing changes
is the same as when using the gitlab-ce repository.
Will you accept merge requests on the gitlab-ce/gitlab-foss project
after it has been renamed?
No. Merge requests submitted to this project will be closed automatically.
Will I still be able to view old issues and merge requests in
gitlab-ce/gitlab-foss?
Yes.
How will this affect users of GitLab CE using Omnibus?
No changes will be necessary, as the packages built remain the same.
How will this affect users of GitLab CE that build from source?
Once the project has been renamed, you will need to change your Git
remotes to use this new URL. GitLab will take care of redirecting Git
operations so there is no hard deadline, but we recommend doing this
as soon as the projects have been renamed.
Where can I see a timeline of the remaining steps?