When Auto DevOps is used for private or internal projects, the registry will not be accessible by Kubernetes after the deploy jobs is over. This prevents the cluster from fetching the image again in the future in case it needs it (scaling, failures, etc).
The solution is to create a Deploy Token that will be used as a permanent access to the registry. This could be done manually, but users using Auto DevOps may not be aware of that.
If a project is private or internal, and Auto DevOps is enabled by default at instance level or specifically for the project, automatically create a Deploy Token with a specific name and read_registry scope that will be used for deployments.
@fabio and @ayufan it's not clear to me why there's a different mechanism in the first place for public vs private/internal projects. It seems to make more logical sense to have a common mechanism, where the registry is available for either type of project, in a secure fashion. I'd need to understand more about the design principles first. I'm thinking there's a more efficient way to do this, but we need more information.
/cc @jritchey can you take a look at this mechanism and comment as well?
How is it different from CI_JOB_TOKEN? Is CI_JOB_TOKEN temporary?
Are deploy tokens restricted to the project level?
Is auto devops restricted to the project level?
Will the autogenerated deploy token expire?
What ways is the autogenerated deploy token exposed to the user?
This autogenerated deploy token is read_only?
It will be available as CI/CD environment variable?
If it is exposed via CI/CD environment variables, are shared runners a concern for secret environment variables? (for instance, should it ever be restricted to project specific runners only)
Where and how does kubernetes store this token?
How is it created?
Where else will it be stored and how is it stored?
@kathyw we need to consider it as a separate case, since public projects also have public read access to the registry. This means that anyone can fetch the docker images without providing any authentication.
This is not true for private/internal projects: in those cases, images cannot be accessed if you don't provide proper credentials.
When we deploy to Kubernetes, the cluster needs to access the app image that is stored on the GitLab registry: for public projects, this can be done without problems since the image is public too. For private/internal projects, we need to provide credentials. Our previous approach was to use a temporary token (CI_JOB_TOKEN), but if the cluster needs to fetch the image again (e.g., because of a restart), the stored credentials will not work anymore since the CI_JOB_TOKEN dies when the deploy job ends.
Deploy tokens solve this problem, by providing a read only permanent token for a specific project. But users still need to know that they have to create this token.
Our proposal here is to automatically create the permanent token if the project is private/internal, so it can be used to deploy without any user interaction.
I hope this is clearer now, otherwise we can schedule a call to dig it more.
How is it different from CI_JOB_TOKEN? Is CI_JOB_TOKEN temporary?
Yes. This one is permanent and can be used even after the job ends.
Are deploy tokens restricted to the project level?
Yes. They allow access only to that specific project.
Is auto devops restricted to the project level?
Not sure what you mean here... could you please explain more?
Will the autogenerated deploy token expire?
No, it is intended to be permanent. It can be manually deleted by the user at any time, anyway.
What ways is the autogenerated deploy token exposed to the user?
Via an environment variable accessible in CI/CD jobs (we can restrict to be exposed only for deploy jobs).
This autogenerated deploy token is read_only?
Yes. Restricted to just read registry images.
It will be available as CI/CD environment variable?
Yes, see above.
If it is exposed via CI/CD environment variables, are shared runners a concern for secret environment variables? (for instance, should it ever be restricted to project specific runners only)
I don't think so. Shared runners guarantee isolation for different projects.
Thanks @bikebilly . I don't see anything glaring as of now. Generally it is best practice to rotate secrets periodically (possibly <=12months in this case). So a token expiration/rotation is something to consider.
I think we should also make the user aware that a (permanent) deploy token will be created for the project, whether it's via documentation or a notification of some sort.
If a project is private or internal, and Auto DevOps is enabled by default at instance level, automatically create a Deploy Token with a specific name and read_registry scope that will be used for deployments.
@bikebilly Why would we only create the token automatically if the Auto DevOps pipeline is being used as a default at the instance level? Wouldn't users face the same problem if the pipeline is being defaulted to at an individual project level?
Is it safe from a security perspective to now close https://gitlab.com/gitlab-org/gitlab-ce/issues/44452 and work on the assumption we are going to create the deploy token automatically? This affects some other issues Im working on so just want to be sure this is the direction we are able to go in.
Should we show any information regarding the token to the user via the UI or is it okay for all of this to happen in the background?
Why would we only create the token automatically if the Auto DevOps pipeline is being used as a default at the instance level? Wouldn't users face the same problem if the pipeline is being defaulted to at an individual project level?
@tauriedavis I totally agree on that, I updated the description to reflect also that case. I'll leave to @danielgruesso to define more details (e.g., at which point the token should be created).
What happens with the username/password that is generated when creating a token if we do it automatically? Should users have the option to revoke the token if we create it automatically?
When auto devops is disabled (ay any level) the token should be revoked.
Since the user shouldn't need to manually call the token I don't think it's necessary to show them any information regarding the token (at least initially, we can iterate).
I can't really think of a use case where the user would want to revoke the token so maybe we don't need to include this in MVC. What do you think @bikebilly@markpundsack
In general, the basic security stance is that tokens should be revokable/rotatable. There's always some vector by which credentials are leaked and rotating credentials is the common way to minimize risk. e.g. someone leaves your team and you rotate the deploy tokens.
Generate a token for all projects regardless of Auto DevOps enablement. (it's useful to have deploy tokens outside of Auto DevOps too, so why restrict this?)
Note that rotating creds can become a rat hole. If you revoke a credential, you then need to propagate the creds to all consumers, which in our case probably means triggering a redeploy, but then there's a window where the system is vulnerable to downtime because deployments take non-zero time, and until the deployment finishes, old pods may die and be respawned and try to use stale creds. Some cred systems thus rotate through having two valid tokens at a time. I think this is beyond scope currently, but it may come up and maybe even something we should proactively think about for the future.
I agree that auto revoke is not needed in this iteration. Since they will be regular deploy tokens, users can revoke them manually in the project settings page.
@mayra-cabrera this makes the creation of the token automatic.
The big difference is that when settings up Auto DevOps on private projects, users still need to know they have to manually create the deploy token. If we create it for them, they will not need to know (or at least, this is not strictly required from a technical point of view, it would just be informative).
@tauriedavis the deploy token is created as an env variable; we don't show env variables in the GUI. If a user wants to view them they have to use debug tracing. I think that suffices, do you agree @bikebilly?
The deploy token should be already visible in the list of deploy tokens. I don't think there is the need to state what is that for, probably the name is enough. If people will complain they have a deploy token they don't understand, we can add some sort of messaging in the deploy token page later.