Build KAS in FIPS mode

added to epic &7933 (closed)

added Category:Kubernetes Management FIPS FIPS Follow-up direction groupconfigure [DEPRECATED] typefeature labels

changed the description

added workflowrefinement label

added workflowready for design label and removed workflowrefinement label

added workflowrefinement label and removed workflowready for design label

Setting label(s) ~"devops::configure" sectionops based on ~"group::configure".

added devopsconfigure [DEPRECATED] sectionops labels

changed milestone to %Backlog

added environmentsbacklog label and removed workflowrefinement label

added [deprecated] Accepting merge requests label

marked this issue as related to gitlab-org/cluster-integration/gitlab-agent#288 (closed)

marked this issue as related to gitlab-org/cluster-integration/gitlab-agent#220 (closed)

Hi! 👋 this issue was created as a placeholder to enable KAS in FIPS mode, KAS has been excluded from the first round of FIPS work and this has been created to start gathering feedback and kickstart the work.

The issue is pretty rough, but we have new guidance (internal) to tackle FedRAMP MilestoneRAR Requirement ASAP.

In particular I am not sure why we removed ~"FedRAMP::Needs PM triage" for such a an unrefined issue, was this intentional or maybe part of a bulk move? If not I just want to surface that we will need to refine and add details to this before it can be worked.

I am pinging @joshlambert given that @connorgilbert and @nagyv-gitlab are on PTO, apologies if it was the wrong ping

/cc @dcroft @sgoldstein

Thanks for this, @nmezzopera.

@joshlambert as you can see from the issue description, this isn't something we can work on until it's been developed into a viable problem definition and solution. Based on this, I'm removing the FedRAMP label as we can't actually deliver anything at this point.

@nmezzopera @dcroft @nagyv-gitlab

In particular I am not sure why we removed ~"FedRAMP::Needs PM triage" for such a an unrefined issue, was this intentional or maybe part of a bulk move? If not I just want to surface that we will need to refine and add details to this before it can be worked.

Sorry that this was confusing. The descriptions of these labels are:

~"FedRAMP::Needs PM triage": "Next step: FedRAMP Product Manager updates, reviews, moves, or closes this issue"
~"FedRAMP::Needs group input": "Next step: product group or subject matter expert reviews issue and identifies next steps"

I, as FedRAMP PM, triaged this need (having KAS work in FIPS mode) into the RAR requirement milestone (since, if we do not have FIPS 140 compliance, we are not allowed to even submit a Readiness Assessment Report (RAR)). You can see some further discussion on the parent epic at &7933 (comment 1047136820).

@joshlambert as you can see from the issue description, this isn't something we can work on until it's been developed into a viable problem definition and solution. Based on this, I'm removing the FedRAMP label as we can't actually deliver anything at this point.

I am reapplying the label because we need to track FIPS 140 compliance for this feature or explicitly document this feature being out of scope and possibly gain approval from our federal authorizing official to proceed on that basis (see related Slack thread from today (internal link)). (To avoid confusion in the future, I've also updated the milestone label descriptions to reflect that they shouldn't be removed without coordination. The labels identify what is we have determined to be required for FedRAMP, so they can't be removed because of a deficiency with a particular issue.)

Have we not yet specifically defined together what it will take to render KAS FIPS-compliant? (I was not PM for FIPS 140, so @dorrino you might know?) In general FIPS 140 compliance means using only validated cryptographic modules, only allowed cryptographic algorithms (e.g. no MD5), and encrypting data in transit and at rest. https://internal-handbook.gitlab.io/handbook/engineering/fips-compliance/ has more details. In general, each product group was to have assessed their own software artifacts and their FIPS compliance posture as part of the FIPS effort (&6452 (closed)).

I will note that mutual TLS (bidirectionally authenticated) is not required (see further discussion in the epic comment: &7933 (comment 1050526218)), though TLS is.

@connorgilbert Thanks. We are trying to prioritize the parent epic ASAP. I'll let you know what is a realistic timeline to deliver it once the planning is done.

Given PTOs and Visiting grants, the refinement/planning will likely take a few weeks.

@connorgilbert Thanks for the update and clear explanation.

Have we not yet specifically defined together what it will take to render KAS FIPS-compliant?

We have not done so explicitly, we did open a serie of issue as part of the parent epic (all of which minus this are closed) KAS was skipped during the first FIPS round and disabled in FIPS builds

I checked the epic &6452 (closed) and I do not see Configure involved anywhere.

The last time we worked on FIPS was at the beginning of the year where KAS was determined to be excluded from the first round. This is why I was confused when this issue suddenly grew in priority. As I mentioned we initially created this as a placeholder to add FIPS compliance into KAS. Had we known this (and apologies if I should have known this and I somehow missed) we would have planned this differently, especially with the PTO (paternity leaves) that we have right now.

I am discussing this because I want to avoid this kind of situation to happen again, I see that @grzesiek is on this now 🙏 and @ash2k Is answering some of the uncertainties (Thanks! 🙇) so we should be good to go.

I am available to offer any assistance necessary!

@ash2k will you be take to take this over from here? I think that we have a good understanding now about what is needed to make KAS FIPS-compliant, right?

removed FedRAMP MilestoneRAR Requirement label

added FedRAMP MilestoneRAR Requirement label

mentioned in epic &7933 (closed)

added workflowrefinement label

changed the description

assigned to @grzesiek

Let me assign to myself for the time being. I would like to learn more about KAS and perhaps figuring it out what would take to make it FIPS compliant would be a good first step. @ash2k seems to be on PTO for the next two weeks, and this is an urgent topic. /cc @dcroft @sgoldstein

I started a merge request that links KAS with FIPS-supported OpenSSL library through BoringCrypto ➡ gitlab-org/cluster-integration/gitlab-agent!737 (closed)

I'm not sure if this is enough to actually make KAS FIPS compatible. I need to spend a bit more time on the research. According to &7933 (comment 1075527171) this may be enough, but I will reach out to domain experts like @ash2k and @nagyv-gitlab to actually learn more about it.

@grzesiek I don't know enough technical details to help you here. Based on the linked issue, this issue is blocked by Support TLS for direct websocket connections in... (gitlab-org/cluster-integration/gitlab-agent#220 - closed)

@tigerwnz will be the first to return from PTO and hopefully he'll be able to help you out in the 2nd half on next week.

It seems that this comment describes the way forward gitlab-org/cluster-integration/gitlab-agent!737 (comment 1084605135)

The fastest thing to do right now since I'm not sure if any of the KAS maintainers are around would be to make Omnibus and CNG build scripts replace go_register_toolchains(version = "1.18.5") with go_register_toolchains(version = "host").

That runs the risk of using a version of Go that isn't fully tested with kas, though.

As I mentioned in &7933 (comment 1084213386), I think we should make that change in KAS itself and modify the KAS builds install the Go compiler. This would make kas consistent with all other GitLab components.

@stanhu Do I understand correctly that we would need to:

Update the gitlab-agent project to make it consistent with all GitLab Go projects, to use local Go compiler instead of installing it through bazel.
Add a pipeline in gitlab-agent that will test it with additional Go version (we usually support two Go version, right?).
Open a merge request in Omnibus to build KAS with version = "host" Go.
Open a merge in CNG project to build KAS with version = "host" Go.

Did I miss something?

@grzesiek If we do the first two items, we shouldn't need to do 3 and 4 because Omnibus and CNG already provide a Go compiler, assuming we update KAS to use version = "host".

Items 3 and 4 could be done if for some reason we didn't want to do 1 and 2.

@stanhu @grzesiek

Update the gitlab-agent project to make it consistent with all GitLab Go projects, to use local Go compiler instead of installing it through bazel.

kas uses the same Go version in all 3 environments - local dev, CI and CNG. This is due to bazel, being a hermetic build system, installing the requested Go version. I haven't specifically investigated this, but to the best of my knowledge kas is the only component that is consistent in that sense. All other projects use potentially different versions - locally devs have a version, CI has potentially a different version and CNG has potentially yet another version. So by making kas use a local compiler we'd make things consistently inconsistent :almostawesome: Omnibus is an exception here, it uses local compiler version and we get problems like this one because of that. We couldn't make it work with bazel when we still supported CentOS 6 or something. We need to fix that.

Having said the above (just wanted to share how I see it), I understand that there is a need and an urgent one. I agree that using the local version for FIPS builds is the way to go at least short term. But we need to ensure that the version that is used in CI is the version that is used in CNG and in Omnibus. For now we can just copy-pase the version and only use the forked compiler when we need it.

Long term the right approach would be to create a custom bazel Go toolchain with the compiler we use (see go_download_sdk) for FIPS and just use that in CI, CNG, Omnibus, and maybe locally too.

What about agentk? Is it in scope for FIPS? It's a container image we publish on release/tag from CI in the agent repo. It is not built by Omnibus/CNG. This is mostly because CNG doesn't support multi-arch images and we need (due to customer demand) to support arm and arm64 archs too.

In &8660 (closed), we're discussing how to streamline the Go upgrade process. I think we're leaning towards a solution that uses images built by Distribution (e.g. registry.gitlab.com/gitlab-org/build/cng/gitlab-go) so that all components will build and test against the current and next supported version of Go instead of having bazel manage the version.

Hm, I'm actually not 100% sure using the host SDK will work the way we expect. This needs to be tested. It may work, I'm just no longer sure.

Another thing about agentk - we currently cross compile for arm and arm64 from the usual amd64 machines. We'll not be able to cross compile if we need CGO. We'll need arm runners. I wonder how Runner solves this problem?