Customer's Jenkins system requires an SSL client key and certificate in order to authenticate. Prior to upgrading to GitLab EE they used a source installation and patched the web hook model to allow this.
If we have this as a proper feature in GitLab there's probably a question about whether it should be a global configuration or where a user uploads the key and cert specific to a given web hook. An initial concern about a global hook could be that the private key and cert would be sent with all web hooks even those that don't require it. This could be a potential security issue, I'm not sure.
Original description
When a webhook gets triggered in our environment, our server (GL) needs to presented to the receiving server (for example Jenkins). We have some code inside that adds it to the model for webhooks:
if Gitlab.config['pem_cert'] and Gitlab.config['ssl_ca_file'] pem File.read(Gitlab.config['pem_cert']), Gitlab.config['pem_cert_passphrase'] ssl_ca_file Gitlab.config['ssl_ca_file']end
This essentially enabled two-way SSL communication (SSL client certs).
The user was originally doing this via GitLab v7.14 (source CE install). They were changing the WebHook model. The corresponding file on a EE omnibus install would be /opt/gitlab/embedded/service/gitlab-rails/app/models/hooks/web_hook.rb. The code block was being placed after the include Sortable line.
The current version of their GitLab installation is 11.7 (EE omnibus).
This has broken several pretty important projects. We NEED this.
What authentication options do users have today
Users can pass a secret token in the HTTP header request to the webhook, which can then be used to verify the request is legitimate.
This is for a large, important, strategic client that has recently upgraded. They were using CE and were manually changing items themselves every time they upgraded, but they upgraded to EE so they could work with us to get these integrated into the product. https://gitlab.my.salesforce.com/0016100001CXGCs
@lmcandrew , This customer is not looking for a code change right now, but wants to know if a workaround is available (even if that is a code change they make in their instance) . This is blocking them from upgrading off of CE to EE
@lyle, is this something your team might be able to help with as well? I'm not sure if I understand the issue, but a developer or support engineer on a call might be exactly what we need.
@kchasse Thanks for tagging me. I'm slightly confused - does the customer have a workaround for CE that doesn't work with EE? Or are they manually altering requests and no longer want to do so when they move to EE?
If I understand correctly, the issue is they need to have GitLab send a private certificate with the request - an SSL client certificate.
Unfortunately, this isn't a feature we currently have with web hooks and it will have to be implemented as a proper feature. For the moment we can probably only help to repatch as they have before.
@kchasse I was helping the customer in support along with @jcolyer. Based on what we found, I updated the issue description to better reflect the feature the customer needs. I'm also adding the stage label so the product manager(s) see the issue and can schedule.
Excellent! By the way, Support was able to help with a workaround, but it will be a painful upgrade for this customer until we get this in a release, so I am pushing for this sooner rather later.
@jeremy we may want to have an internal discussion (and possibly one with the customer) to discuss this. This might be really unique to this customer, but any solution should be usable by others.
Let me chime in from the customer perspective. This is a case of modified code in a previous version of GitLab to allow this to happen. We've committed to no longer modifying the source code in favor of Omnibus.
The general use case here is that we have users using our GitLab that might have another service set up which uses 2-way TLS (Jenkins being a common case, but not the only case). When a webhook gets triggered, GitLab needs to make a call out the external service and present it's cert/key for a 2-way TLS handshake. 2-way TLS is a common paradigm for us, and I've heard of companies that use it too.
Configure SSL client certs on a hook-by-hook basis
Configure a site-wide SSL client certs (as done here)
The second seems more straightfoward and mirrors the SSL server certs used by GitLab. We would need to add Omnibus support and ensure a copy of keys were installed on each node in a multi-node configuration.
The first case gets a little tricker. We could be talking about hundreds, possibly thousands of certs. This seems like we'd want to integrate with some key management system (e.g. Vault in https://gitlab.com/gitlab-org/gitlab-ce/issues/40720).
Agreed 1 seems complicated, and I'm probably missing some background there.
Re: 2 - That meets our needs. GitLab just needs to be able to say "I'm GitLab" to the external service. The external service is looking for a cert that is signed by a CA it trusts (as is GitLab).
Seems like 2-way TLS could use a little love in GitLab - it needs to be more than requiring a cert in the Nginx config.
Seems like 2-way TLS could use a little love in GitLab - it needs to be
more than requiring a cert in the Nginx config
What do you mean? The current Nginx configuration isn’t meant to
handle client authentication. It’s only meant for HTTPS. For client
authentication we need to add something to GitLab Rails itself for outbound
web hooks. That’s what Stan is proposing in his 2 options. And it sounds
like number 2 works for you.
Just a quick note. This is a blocker for connecting GitLab and Jenkins on OpenShift. OpenShift will self-sign a key for Jenkins install. There are ways to make a key for Jenkins on OCP with a CA but GitLab needs to recognize and support it.
@dblessing - that was a somewhat off-the-cuff comment, which comes from a desire for GitLab to work more cleanly out of the box with the various applications of 2-way SSL that I use. As @cdmaurer13 said, there should probably be some further discussion about this. You mentioned that the Nginx configuration isn't meant to handle client authentication, but I would argue it should be because that is how I have to use it. In order to explain everything, I'd have to write a book, so I'll leave that in depth explanation for the referenced discussion.
@emosher I may not be understanding, but Nginx can't handle outbound client authentication. Nginx isn't even in the mix when GitLab makes an outbound call such as with a web hook. The connection initiates from within GitLab Rails and goes out to the configured endpoint, without any interaction with Nginx.
Are you referring to some other use-case for client auth where Nginx would be in the mix - such as for other systems authenticating to GitLab, which would be inbound? This issue is about outbound calls, specifically web hooks, so I don't want to get us off-track, but I hope I'm understanding correctly.
No, I think you are understanding correctly. The larger conversation I mentioned includes both inbound and outbound calls. This issue solves the outbound problem for webhooks. Apologies for getting off topic for this issue.
@jeremy the workaround is no longer working consistently for this customer -- we may need to look at assigning an upcoming release and/or re-engaging with support on another workaround.
@kevinchasse: Why is the workaround no longer working? Do we have any details? I'm wondering if the quickest step is to get that unblocked if a recent change was responsible.
I am trying to get more info, but here is what I have:
Prior to COVID we were on 12.8.x and things appeared to be working. It seems like things went bad when we upgraded to 12.9.x then to 12.10.x shortly after. Attempts to make the patch work succeeded only temporarily.
I'm not certain why the code stopped working. It is the same as it was before the upgrade(s) and applied it in the same manner (change files, gitlab-ctl restart).
@g.hickman This came up on a dashboard for Auth features. I am not sure I even understand it enough to formulate a question, but, is this a request you've heard? Yes, because it says "Webhooks" you're getting involved
This has come up again as the patch provided to the customer no longer works as of 16.6 per reports from the customer via https://gitlab.zendesk.com/agent/tickets/117177 (Internal, US Citizen access only)
@stanhu customer implemented the workaround but is seeing strange behavior:
When we test a webhook with SSL Verification enabled the test works fine, but if we uncheck SSL Verification, we receive a 400 error saying that the certificate verification failed.