When the pods are created, they cannot pull images in the GitLab container registry or any other private container registry.
Creating a secret and editing the deployment right after the workspace is created on gitlab with "imagePullSecrets: gitlab-registry-credentials" does make it work. But this is a very hacky solution and the changes have to be done constantly since the reconciliation between agent and rails will reset this change.
Using managed Kubernetes services like EKS does not particularly make it easy or potentially even feasible to add the credentials at the node level.
One might expect this to 'just work' with the credentials created for the workspace.
Perhaps a solution is to create the workspace token also with registry read access and before creating the Deployment for the workspace, also create a secret using that token referencing registry.gitlab.com in the workspaces namespace and also adding the 'imagePullSecrets' to the deployment spec.
Acceptance Criteria
TODO: Fill out (required)
[Describe what must be achieved to complete this issue.]
[Describe another requirement needed to complete this issue.]
[Add additional acceptance criteria as needed.]
Technical Requirements
TODO: Fill out or delete [If applicable, please list out any technical requirements for this feature/enhancement.]
Design Requirements
TODO: Fill out or delete [If applicable, please provide a link to the design specifications for this feature/enhancement.]
Impact Assessment
TODO: Fill out or delete [Please describe the impact this feature/enhancement will have on the user experience and/or the product as a whole.]
User Story
TODO: Fill out or delete [Provide a user story to illustrate the use case for this feature/enhancement. Include examples to help communicate the intended functionality.]
I wasn't sure if this was the best place to post this, not sure if it's actually handled in the proxy or the agent but I've also mentioned this in the workspaces feedback issue - along with providing a workaround for this problem: #410031 (comment 1589792849)
Hi @cms we've released support for private container registries in 17.6 (see documentation here). We’d love to hear the user's feedback to ensure it meets their needs and identify any improvements. Let me know if they’re available to provide input or if there’s anything we can do to support them in using it
Configure image pull secrets in agent configuration and replicate it in each workspace's namespace
During setup of Kubernetes cluster, administrator will create a image pull secret. Let's say they created this in a secret called example-image-pull-secrets in default namespace.
Configure the agent configuration to specify this information. You can specify multiple image pull secrets.
During workspace creation, when a new namespace is created in Kubernetes, agent will clone these secrets into the new namespace.
Configure the Kubernetes Deployment to use the secrets in its namespace to pull the images from private registry.
Pros
Works for registries other than GitLab as well.
Generic enough that does not get into the complicated permissions model that would be different for each organization - not every user might get
Cons
One additional administrative step is required - to create the image pull secrets during setup and configure it in agent configuration.
Use the PAT generated on workspace creation to pull images
During workspace creation, we generate a PAT. The primary purpose of this PAT right now is to interact with GitLab (git operations).
Increase the permissions of this PAT to include read_registry.
During workspace creation, create an image pull secret in the namespace of this workspace.
Configure the Kubernetes Deployment to use the secrets in its namespace to pull the images from private registry.
Pros
No additional administrative steps are required.
Cons
Does not work for other registries apart from GitLab.
Can we run into some complicated permissions model which are different for each organization - maybe not every developer(e.g. contractors) has access to the container registry.
My opinion is that we start with Configure image pull secrets in agent configuration and replicate it in each workspace's namespace. And if there is enough customer demand/requests, then look into another approach - Use the PAT generated on workspace creation to pull images.
My opinion is that we start with Configure image pull secrets in agent configuration and replicate it in each workspace's namespace. And if there is enough customer demand/requests, then look into another approach - Use the PAT generated on workspace creation to pull images.
@vtak based on the pro/con list I like your suggestion because it aligns with the design target of being flexible/agnostic whenever possible. Giving extra attention to the UX of the admin step could potentially help reduce the downside here as well.
I might have a knowledge gap here but unless we have a specific requirement to use the PAT or the admin step is extremely burdensome, increasing coupling with GitLab's registry and dealing with unknown troubles around fitting into each org's unique permission model seems like an expensive tradeoff.
Add the support for image_pull_secrets to the agent configuration (and make it optional, which I assume we would be doing anyway).
If image_pull_secrets is not specified for a workspace's agent, then add the read_registry permission to the workspace's PAT.
This would still allow flexibility for other registries, but also allow it to work "out of the box" with GitLab registries with no extra entry in the agent config.
While I don't have any strong objections to doing both of them, I'm leaning towards, let's first provide one solution and see how much demand is there for the another one. If the solution that we provided solves for most, if not all, use-cases, then we can postpone the other solution.
While I don't have any strong objections to doing both of them, I'm leaning towards, let's first provide one solution and see how much demand is there for the another one. If the solution that we provided solves for most, if not all, use-cases, then we can postpone the other solution.
I understand that this is in the spirit of iteration, but I also think the points made above are compelling enough to justify doing both of these at once, so we can have all the pros and none of the cons.
I think the second option may actually be the better initial option.
In addition to the extra steps required to create the secrets and configure the agent I'm pretty sure the first option requires an agent administrator to expose all possible registry secrets to any workspace user since the pods being launched will have access to the kubernetes secrets.
The second option ensures that no credentials are being exposed in a surprising fashion. An additional option here may be to leverage CI Variables or Secrets Manager to create the registry credentials on a per project basis. This still results in shared credentials, but the scope would be substantially more limited.
I'm pretty sure the first option requires an agent administrator to expose all possible registry secrets to any workspace user since the pods being launched will have access to the kubernetes secrets.
Isn't that the point, though? In order to allow workspaces to pull the images, they have to have access to the secrets to do so. Or am I missing something?
In addition to the extra steps required to create the secrets and configure the agent I'm pretty sure the first option requires an agent administrator to expose all possible registry secrets to any workspace user since the pods being launched will have access to the kubernetes secrets.
The second option ensures that no credentials are being exposed in a surprising fashion.
No secrets are visible to the end user anyway. Of course, any one with Kubernetes secret access can view them but that is by design.
My biggest problem with going with the second approach, IMHO, is that it targets a smaller subset of audience. Our customers are large and varied. They use many different container registries. Our solutions should reflect the same.
My biggest problem with going with the second approach, IMHO, is that it targets a smaller subset of audience. Our customers are large and varied. They use many different container registries. Our solutions should reflect the same.
We can do them in separate small iterations, but since they both should be only backend / agentconfig changes changes and associated documentation, with no (AFAIK) UI changes needed, then there's no reason we shouldn't do them both while we are working on this track of work, and ensure that we can meet this feature's requirement for all use cases and users.
@vtak The problem with using the agent configuration is the imagePullSecret gets created in a way that is accessible to all projects under the top level group. The serviceAccount the workspace pod runs as must be able to access it so that the pod can be created. When the pod is created the credentials for that serviceAccount are stored in /var/run/secrets/kubernetes.io/serviceaccount so any pod launched from any project can use those serviceAccount credentials to read all the secret it has access to from the API.
The imagePullSecret and serviceAccount should be scoped to the Project level and the serviceAccount restricted as much as possible to prevent escalation or lateral movement.
@vtak I would have to test it to be sure, but I'm 99% certain that a pod manifest needs to have a serviceAccount that can read the Secret containing the creds for the image pull or it won't be able to pull the image. Otherwise anyone that can create a pod can pull from any repository a cluster has a secret for. I'm also pretty sure the default behavior is that the Secret also has to be in the same namespace as the pod.
Some users have complained about the workaround itself being not stable in #441226 (comment 1899259057) . We might want to priroitize this issue to provide first-class support. We can of course decide on the initial scope.
Hey team, I put together a workshop for a large GitLab Premiumcustomer(SFDC Link) who was keen to leverage Workspaces as a way to manage their development environments in the same place as their code. To do this, I tried to demonstrate managing a development tooling container in CI in the same place as the project when I ran up against this issue. Commenting to capture the use case and interest.
Nice workshop @bridley . Would be great to see a blog out of this as well
As for the use-case currently, you can unblock yourself with a workaround mentioned in #441226 (comment 1843510748) until we get to providing first class support.
Hi @bridley we've released support for private container registries in 17.6 (see documentation here). We’d love to hear the user's feedback to ensure it meets their needs and identify any improvements. Let me know if they’re available to provide input or if there’s anything we can do to support them in using it
@katrinleinweber-gtlb we've released support for private container registries in 17.6 (see documentation here). We’d love to hear the user's feedback to ensure it meets their needs and identify any improvements. Let me know if they’re available to provide input or if there’s anything we can do to support them in using it
Hi @ckaburu! I'm providing an update here that we are starting work on this next. This is the epic here Provide support to use images from private cont... (&14664 - closed) which you and the customer can use to follow along for progress. I will update here again once we are closer to release. Let me know if you have any further questions!
Hi @ckaburu we've released support for private container registries in 17.6 (see documentation here). We’d love to hear the user's feedback to ensure it meets their needs and identify any improvements. Let me know if they’re available to provide input or if there’s anything we can do to support them in using it
Hi @kballon ! I'm providing an update here that we are starting work on this next. This is the epic here Provide support to use images from private cont... (&14664 - closed) which you and the customer can use to follow along for progress. I will update here again once we are closer to release. Let me know if you have any further questions!
Hi @kballon we've released support for private container registries in 17.6 (see documentation here). We’d love to hear the user's feedback to ensure it meets their needs and identify any improvements. Let me know if they’re available to provide input or if there’s anything we can do to support them in using it
Chad Woolleychanged title from Provide first class support for use images from private registries to create a workspace to Provide first class support for use images from private container registries to create a workspace
changed title from Provide first class support for use images from private registries to create a workspace to Provide first class support for use images from private container registries to create a workspace
FYI, I updated the name of this issue to make explicit that we are talking about container registries, and not any other type of private registry support.
@vtak I see you already did some investigations for this. Do you have enough information to refine this into an epic now? If so can we get this done please.
The solution I'm going forward with is Configure image pull secrets in agent configuration and replicate it in each workspace's namespace mentioned in #441214 (comment 1881691817)
Michelle Chenchanged title from Provide first class support for use images from private container registries to create a workspace to Feedback: Provide first class support for use images from private container registries to create a workspace
changed title from Provide first class support for use images from private container registries to create a workspace to Feedback: Provide first class support for use images from private container registries to create a workspace
Why interested: Using Fargate with AWS and looking for a way to specify a namespace at workspace creation rather then use the auto generated one.
Problem they are trying to solve: Currently, whenever I create a new workspace, it creates a namespace and pod with a random name, which results in the pod being unscheduled because the Fargate profile isn't updated with that random name.
Current solution for this problem: Currently they can use a replicator for kubernetes but there is a race condition that makes that usage not ideal.
Impact to the customer of not having this: Poor experience or inability to use Workspace features
Hi @malmendarez we've released support for private container registries in 17.6 (see documentation here). We’d love to hear the user's feedback to ensure it meets their needs and identify any improvements. Let me know if they’re available to provide input or if there’s anything we can do to support them in using it
Hi @onemoz! We are starting work on this next. This is the epic here Provide support to use images from private cont... (&14664 - closed) which you and the customer can use to follow along for progress. I will update here again once we are closer to release. Let me know if you have any further questions!
We have a GitLab UltimateUS Gov customer that would like to be able to use private container registries with GitLab Workspaces. They would like the ability to configure what is used for the init containers when the workspace stands up, so that they can point to a different registry for the git container that clones the code. And the VSCode container that sets up VSCode in the "main" workspace container.
Hi @iHeartLinux we've released support for private container registries in 17.6 (see documentation here). We’d love to hear the user's feedback to ensure it meets their needs and identify any improvements. Let me know if they’re available to provide input or if there’s anything we can do to support them in using it
First class support for private container registries has been added to Workspaces. This includes not just GitLab registries but any kind of registries. Till the time the Kubernetes cluster has a valid image pull secret, you can refer it in the GitLab agent configuration for workspaces which will be used to pull the container images.