troubleshooting procedure for x509 / TLS signed commits

Problem to solve

Support is assisting a customer with x509 / TLS signed commits (GitLab team members can read more in the ticket)

They are seeing the commits as unverified.

To troubleshoot this, we need to crack:

1. Exactly which certificate authority certificates need to be trusted for this to work. This is likely to be almost exclusively used with private CAs, so cannot count on using pre-existing trust of public CAs.
2. How do we walk through the internal checks done by GitLab to determine verified status:

      def verification_status
        return :unverified if
          x509_certificate.nil? ||
          x509_certificate.revoked? ||
          !verified_signature ||
          user.nil?

The customer's supplied support with

• A test repo
• CA certificates

So, it ought to be possible to manually check all these steps, with the exception of user.nil? - which requires their email address in an account in our lab GitLab. To troubleshoot this specific customer's issue we can remark out the code, customers will be able to follow the same documented steps without needing to do this.

1: TLS

It's not clear to me how to get the TLS stack to work correctly for this feature.

When GitLab acts as a client (eg: connecting out to Elastic Search or JIRA, or any other external server) the procedure is to put the root certificate authority in GitLab's trusted certificates.

The server should then provide its leaf certificate, the intermediate certificate for the CA that signed the leaf certificate, and any other intermediates, up to the one signed by the root.

Here's an example of a multi-tier CA structure, where the websites have to provide two intermediates:

corporate root CA -> administrative CA1 -> website certs CA1 -> website certificates
corporate root CA -> administrative CA2 -> website certs CA2 -> website certificates

GitLab has to root .. trust is established.

How does it work with signed commits?

In my example, the company also has six dedicated CAs for user certs (Smartcards)

corporate root CA -> administrative CA1 -> user certs CA1, CA3, CA5 -> user certificates
corporate root CA -> administrative CA2 -> user certs CA2, CA4, CA6 -> user certificates

The signing itself must be done by user's private key, but that has no identity in it, it's just a key.

• I assume their certificate is included as part of the signing process. Their certificate has email address SAN(s) it has a validity period etc.
• Are intermediates includes as well during signing? Should they be? If not, how is GitLab supposed to build a chain back to the corporate root?

---

The next question is, how do we validate the TLS chaining is correct. With websites, we use openssl s_client to query the web server, and can specify where to find the roots, or a specific root, and so on. And inspect the output.

How do we do this with a signed commit that's buried in GitLab? This may be a moot question if we can do it with ..

2: Rails

I've got this far.

• snag the signed commit

project=Project.find_by_id('121')
c=project.repository.commit_by(oid: '87fdbd0f9382781442053b0b76da729344e37653')

• I have an object with attributeS:

irb(main):010:0> pp c.sha
"87fdbd0f9382781442053b0b76da729344e37653"

• from here

irb(main):010:0> pp X509CommitSignature.by_commit_sha(c.sha)
[#<X509CommitSignature:0x00007f312ef9dab0
  id: 1,
  created_at: Mon, 14 Jun 2021 08:15:56 UTC +00:00,
  updated_at: Mon, 14 Jun 2021 08:15:56 UTC +00:00,
  project_id: 121,
  x509_certificate_id: "[FILTERED]",
  commit_sha: "87fdbd0f9382781442053b0b76da729344e37653",
  verification_status: "unverified">]

• rooting around ..

repo = c.project.repository.raw_repository
@signature_data = Gitlab::Git::Commit.extract_signature_lazily(repo,c.sha)

• and I find I have an array

irb(main):045:0> pp @signature_data[0]
"-----BEGIN SIGNED MESSAGE-----\n" +
"(signed message - base64)\n" +
"(signed message - base64)\n" +
irb(main):044:0>  pp @signature_data[1]
"tree 091d72fac91678ccc91da147d51771ff63e395c2\n" +
"author user <user@example.com> 1621324428 +0200\n" +
"committer user <user@example.com> 1621324428 +0200\n" +
"\n" +

• so I have the signature in [1] but I couldn't future out the ruby gymnastics needed to start at it, with the goal ultimate to manually crank these tests.

          x509_certificate.nil? ||
          x509_certificate.revoked? ||
          !verified_signature ||

• all I know is that at least one of them has failed.
• It'd also be nice, for example, to be able to display the payload (on the rails console) using the openssl libraries along the lines that openssl x509 would do on the command line.

Proposal

Once we have the pieces of the puzzle, I'd propose documenting it.