Jenkins integration should adhere to ci.skip push option

Proposal

Currently the Jenkins integration will execute even if the ci.skip push option is specified. Considering Jenkins is an alternative CI engine that GitLab readily integrates with it should behave in a consistent manner.

When changes are pushed from a local repository to GitLab with the ci.skip push option then the Jenkins integration should not trigger the webhook call.

This issue was automatically tagged with the label ~"group::integrations" by TanukiStan, a machine learning classification model, with a probability of 0.85.

If this label is incorrect, please tag this issue with the correct group label as well as automation:ml wrong to help TanukiStan learn from its mistakes.

If you are unsure about the correct group, please do not leave the issue without a group label. Please refer to GitLab's shared responsibility functionality guidelines for more information on how to triage this kind of issues.

Authors who do not have permission to update labels can leave the issue to be triaged by group leaders initially assigned by TanukiStan

This message was generated automatically. You're welcome to improve it.

added automation:ml groupintegrations [DEPRECATED] labels

added devopsmanage sectiondev labels

Hey @eamell_msi, thank you for creating this issue!

To get the right eyes on your issue more quickly, we encourage you to follow the issue triage best practices.

PLEASE NOTE: You will need to use the @gitlab-bot label command to apply labels to this issue.

To set expectations, GitLab product managers or team members can't make any promise if they will proceed with this. However, we believe everyone can contribute, and welcome you to work on this proposed change, feature or bug fix. There is a bias for action, so you don't need to wait. Try and spin up that merge request yourself.

If you need help doing so, we're always open to mentor you to drive this change.

This message was generated automatically. You're welcome to improve it.

added automation:self-triage-encouraged label

GitLab Premium customer is interested in this feature. Zendesk (internal only)

added backend devopscreate groupcode review labels and removed devopsmanage groupintegrations [DEPRECATED] labels

added customer label

added typebug label

FYI @tlinz @sean_carroll I labeled this ~group::integrations, but it's push option related so it could also be groupsource code.

@tlinz @sean_carroll I agree, as far as I understand it, the problem is with push option, especially having a look at debugging @slynch_ did. I will reassign to your group. Could you please confirm that's correct?

added groupintegrations [DEPRECATED] typefeature labels and removed groupcode review typebug labels

added devopsmanage label and removed devopscreate label

This is quite an import feature for a CI/CD system for it to be missing. We'd like this to be fixed as soon as possible since it forces us to put the logic for skipping a build into the build itself. That process is tedious, fragile, and error prone - not to mention, a waste of resources for build to run when it's not needed.

In the meantime, can you provide any type of workaround to get the Gitlab to see this option?

I did some debugging on this issue and verified that this bug is actually generic and has nothing to do with Jenkins.

I setup the following test:

• I made a simple web server in Node.js. This acted like Jenkins to receive the web hook.
• This web server was deployed in a VM that has network connectivity between it and the server running Gitlab.
• I made a test repo in gitlab and added a web hook with the URL of the VM running the web server.
• I then made some commits to the repo and pushed those commits to Gitlab.

With the web server I could verify when it received the web hook triggered from the push event.

After the first commit, I pushed it to Gitlab using this git command:

git push origin

My Node.js web server received this JSON body from Gitlab. Note that this has the "object_kind" set to "push".

For the second commit, I pushed it with the following:

git push origin --push-option=ci.skip

In this case the expected behavior is to not receive the web hook at all. But the web server did receive the web hook from Gitlab. It had this JSON body. You can see in this case that JSON body is completely different from the first case. And it seems notable that the "object_kind" is set to "pipeline" instead of "push".

This behavior seems to show that if Gitlab sees the push_option, instead of handling it like a normal web hook, it handles it like a Gitlab pipeline.

added groupsource code label and removed groupintegrations [DEPRECATED] label

added devopscreate label and removed devopsmanage label

added typebug label and removed typefeature label

Setting label(s) Category:Source Code Management based on groupsource code.

added Category:Source Code Management label

This behavior seems to show that if Gitlab sees the push_option, instead of handling it like a normal web hook, it handles it like a Gitlab pipeline.

This issue needs investigation. Marking as a spike

changed milestone to %16.1

added priority2 severity2 spike labels

@sean_carroll , do you have any updates on this Issue that we can provide to Juniper Networks at this time?

@pcorpus this is currently planned for %16.1, will that be acceptable to the customer?

Please note it is a spike which means investigation issue. If there are any additional reproduction steps, public repo or other resources that will help us understand the issue, that would be very helpful.

It is not clear if this is a bug or a feature request. Does the GitLab documentation indicate anywhere that should be behaving differently than current?

CC @tlinz @pcorpus

mentioned in issue create-stage#13149 (closed)

added CISP3 label

Moving to %16.2 due to lack of capacity in groupsource code /cc @tlinz @sean_carroll

changed milestone to %16.2

mentioned in issue gitlab-org/quality/quality-engineering/team-tasks#1759 (closed)

@sean_carroll, @tlinz and @a_mcdonald,

This groupsource code bug has at most 50% of the SLO duration remaining and is an SLONear Miss breach. Please consider taking action before this becomes an SLOMissed in 29 days (2023-06-15).

changed due date to June 15, 2023

added SLONear Miss label

set weight to 2

Estimating backend-weight2

Assigned to @j.seto as a %16.1 Deliverable

changed milestone to %16.1

added Deliverable label

added workflowready for development label

assigned to @j.seto

Setting health status to on track as the milestone has just begun.

Issue participants are welcome to override this by setting the health status to another value.

changed health status to on track

added workflowin dev label and removed workflowready for development label

I tested out pushing to a project with the jenkins integration and a generic webhook both configured to be triggered by all applicable and triggerable events.

• Pushing to the project will trigger push hooks for both the jenkins integration and the generic webhook regardless of whether or not the ci.skip push option is provided
• When the ci.skip push option is provided, gitlab seems to create a pipeline run that is skipped immediately which triggers the pipeline event hook.
  • From what I can tell this webhook is meant to be fired when a pipeline is saved so the webhook aspect of this behaviour seems correct. Creating a pipeline run is unexpected (at least to me) and maybe a bug?
  • The pipeline event is separate from the push event and both are triggered by the same push

Should we support a push option for skipping push events for jenkins specifically? or perhaps integrations in general? (e.g skip_push_event_hook or maybe jenkins.skip_push) @tlinz

• Reusing ci.skip might be unwieldy if a project is using both jenkins and gitlab CI/CD
• I think it should be a similar amount of work either way (we control what hooks to execute around here: https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/services/git/base_hooks_service.rb#L64)

I haven't looked into how exactly but I think a practical workaround for the moment would be to configure jenkins to ignore push events based on commit messages and include something like ci-skip in the commit message. There is a push_options attribute in the event payload but it does not actually contain the push_options as expected(#18049).

Hi @j.seto, thanks for looking into this. I think your proposal of a generic skip_push_event_hook sounds very reasonable to me. However, let me involve @igor.drozdov for his opinion.

I wonder why we create a pipeline run and then skip it rather than not creating it in first place when ci.skip push option is triggered. Do you understand this? Would it be an alternative to change that behavior and then use ci.skip to not create pipeline run just like it shouldn't create Jekins run? I suspect that change would have more complex implications than your proposal. @carolinesimpson and @jreporter, do you know why the pipeline run is created?

cc: @sean_carroll

Just to clarify, the pipeline event is not one of the triggers for the Jenkins integration hooks. The pipeline event would be for projects that configured generic webhooks for that event.

@tlinz - looks like this a known bug and was an implementation issue #28369

@j.seto @tlinz according to #403065 (comment 1350552901), when ci.skip is passed, only the pipeline event is received. In this case, I believe fixing Passing [ci skip] now creates a pipeline and sh... (#28369) would likely resolve the issue. But as far as I understand, during our own investigation two events are triggered, right?

Should we support a push option for skipping push events for jenkins specifically? or perhaps integrations in general? (e.g skip_push_event_hook or maybe jenkins.skip_push)

If we decide to choose this option, I'd vote for something general. However, I wonder if there's a way to reuse ci.skip since it would be a straightforward experience for a user.

• Reusing ci.skip might be unwieldy if a project is using both jenkins and gitlab CI/CD

If we don't trigger push event when ci.skip is specified, I wonder if that's also a problem when there's an integration that is not related to CI and it still expects push event to be triggered?

---

I'm not very familiar with how integrations work, just thinking out loud here:

• I see supported_events defined for integrations
• I expect that when push merge_request tag_push event is triggered, then a web hook for the integration is executed
• I wonder if we could rely on pipeline event instead in this case? So if Passing [ci skip] now creates a pipeline and sh... (#28369) is fixed and doesn't create a pipeline, and doesn't trigger pipeline event, then a web hook won't be executed and a Jenkins pipeline won't be created.

@j.seto do you think it may work? or am I missing anything here?

@m_frankiewicz could you please check or help to find someone who could check this reasoning?

If we don't trigger push event when ci.skip is specified, I wonder if that's also a problem when there's an integration that is not related to CI and it still expects push event to be triggered?

@igor.drozdov Now that you've mentioned it, skipping push events in general is also unwieldy. Maybe having ci.skip prevent push events for the jenkins integration specifically is the better approach.

ah, I see that @m_frankiewicz is OOO, let's try to ping an engineer then

@.luke could you please have a look at the proposed solution here: #403065 (comment 1406254547)? would it work, WDYT?

Updated documentation to reflect the fact that the ci.skip option does not apply to integrations (!123535 (merged)).

Created a follow up issue to track the work of implementing a new push option.

I wonder if we could rely on pipeline event instead in this case? So if Passing [ci skip] now creates a pipeline and sh... (#28369) is fixed and doesn't create a pipeline, and doesn't trigger pipeline event, then a web hook won't be executed and a Jenkins pipeline won't be created.

@igor.drozdov Sorry for the late reply! The Jenkins integration is triggered currently for commit pushes, tag pushes, and merge request events. I suspect some people might be using Jenkins to do something specific when, say, tags are pushed, rather than when a pipeline is run, so I don't think we can replace those event hooks with pipeline.

I'll add some ideas to the new issue #415416 (closed).

added workflowin review label and removed workflowin dev label

marked this issue as related to #28369

I'm not sure if this needs docs, @j.seto. If yes, could you add documentation to the labels?

added twtriaged label

mentioned in issue gitlab-org/quality/triage-reports#12804 (closed)

added documentation label

mentioned in commit ed9a5453

mentioned in merge request !123535 (merged)

mentioned in commit 85e3c999

mentioned in issue #415416 (closed)

closed

marked this issue as related to #415416 (closed)

mentioned in issue gitlab-org/quality/quality-engineering/team-tasks#1993 (closed)

Hey guys,

I'm seeing a weird issue with my pipelines and could not figure out if it was a bug at my end or is related to this issue.

When I try to push changes from my local machine to my GitLab Project using git push origin --push-option=integrations.skip_ci, the Jenkins Integration triggers a Push Hook but the Jenkins Pipeline does not get triggered. This works for me.

However, when one of my Jenkins CI/CD jobs running a Python script tries to push changes to the same GitLab Project using the same command git push origin --push-option=integrations.skip_ci (via subprocess in Python), the Jenkins Integration triggers a Push Hook and the same Jenkins Pipeline does get triggered.

I cannot seem to figure out why!

Can someone please help? Would really appreciate it.