## Summary Add instance-level SSH certificate support that is managed through the database and exposed via the Admin UI/API, instead of file-based config. The file-based version shipped in [gitlab-shell#841](https://gitlab.com/gitlab-org/gitlab-shell/-/work_items/841). It lets self-managed admins configure trusted CAs via `sshd.trusted_user_ca_keys` in `config.yml`. Dedicated tenants cannot use it, since they have no access to `config.yml`, `gitlab.rb`, or `gitlab.yml`. This epic covers a DB-backed alternative so instance-level SSH cert auth works on Dedicated, and on self-managed, without touching the filesystem. This is the instance-scoped counterpart to [group-level SSH certificates](https://docs.gitlab.com/user/group/ssh_certificates/) (&10662). CAs live in the DB and are resolved by gitlab-sshd via the internal `/authorized_certs` endpoint. ## Background GitLab currently supports SSH cert auth three ways: 1. **OpenSSH, instance-level.** Configured via `sshd_config` with `TrustedUserCAKeys` and `AuthorizedPrincipalsCommand`. Requires manual config and filesystem access. 2. **Group-level, DB-backed.** CAs are stored per top-level group. They're managed via the [REST API](https://gitlab.com/help/api/group_ssh_certificates) and resolved through `/authorized_certs`. Available on Premium/Ultimate, GitLab.com only. 3. **Instance-level, file-based (gitlab-sshd).** Shipped in [gitlab-shell#841](https://gitlab.com/gitlab-org/gitlab-shell/-/work_items/841) / [!1396](https://gitlab.com/gitlab-org/gitlab-shell/-/merge_requests/1396). gitlab-sshd loads CA keys from `sshd.trusted_user_ca_keys` at startup. It grants instance-wide access, with no namespace, using `cert.KeyId` as the username. None of these work for Dedicated. Option 3 needs `config.yml` edits and a restart. It also requires keeping the file in sync across every gitlab-sshd node. Dedicated tenants only have the Admin Area available to them. The customer asked directly whether this could ship inside Rails itself, the way the group-level feature already does. The gap is that there is no way for a Dedicated admin to set up instance-wide trusted CAs. Group-level CAs are namespace-scoped. On GitLab.com they also require Enterprise Users. Neither covers an instance-wide trust model. ## Requirements - **Offering:** Dedicated and self-managed. Not GitLab.com. There is no instance admin area on GitLab.com, and `.com` customers already have group-level CAs for Enterprise Users. - **Tier:** all tiers. This matches OpenSSH instance-level certs and is not restricted to Premium/Ultimate. - **No filesystem access.** Everything must be manageable via Admin UI and Admin API. - **Storage.** A new dedicated DB table, not the group SSH certificate table. - **Resolution.** Extend `/authorized_certs` to resolve instance-level CAs. On a match, it should return the username with no namespace, matching current file-based behavior. - **Identity mapping.** Certificate `KeyId` maps to the GitLab username. Standard `/allowed` checks apply afterward. - **Precedence.** File-based CAs are checked first. DB-based CAs are checked only when the signing CA is not locally trusted. Both must coexist. - **Validation.** Reject expired certificates, host certificates, and invalid `KeyId` values. These are the same checks gitlab-sshd already applies. - **Revocation.** Removing a CA revokes trust immediately. - **Multiple CAs.** Support rotation and multi-CA setups. - **Auditing.** Emit audit events for CA add/remove. Ideally also emit events for certificate-based authentications, since the file-based approach has no audit trail today and Dedicated compliance expects one. - **Docs and release post.** ## Proposal 1. **Blueprint.** Write a short architecture doc covering the data model, the resolution flow, precedence versus file-based and group-level CAs, and the instance-wide permission semantics. Model it on the group-level blueprint. 2. **Database table and model (Rails).** Add a new table, for example `instance_ssh_certificates`, with `title`, `key`, `created_at`, and appropriate uniqueness and indexing. Mirrors group-level [#421537](https://gitlab.com/gitlab-org/gitlab/-/work_items/421537). 3. **Extend `/authorized_certs` (Rails).** When the signing-key fingerprint matches an instance-level CA, resolve to a user with no namespace. Keep group-level resolution unchanged, and document the precedence between the two. Mirrors [#421538](https://gitlab.com/gitlab-org/gitlab/-/work_items/421538). 4. **Admin REST API (Rails).** Add GET/POST/DELETE endpoints for instance SSH certificates, restricted to admins. Mirror the [group SSH certificates API](https://gitlab.com/help/api/group_ssh_certificates). Mirrors [#421915](https://gitlab.com/gitlab-org/gitlab/-/work_items/421915). 5. **Admin UI (Rails).** Add an Admin Area section to add, list, and remove trusted instance CAs. This is the piece Dedicated actually needs. 6. **Audit events (Rails).** Emit events for CA create/delete, and evaluate whether to also emit auth-time events. Mirrors [#427413](https://gitlab.com/gitlab-org/gitlab/-/work_items/427413). 7. **gitlab-sshd verification (gitlab-shell).** Confirm that `authorizedCertsClient.GetByKey` handles instance-level, no-namespace results correctly. Confirm file-based CAs are still checked first, and review the `FF_GITLAB_SHELL_SSH_CERTIFICATES` guard. This is expected to be a small change or no change at all. 8. **Security review (Rails).** Review the instance-wide trust model, the `KeyId`-to-user mapping, and admin authorization. Mirrors [#428236](https://gitlab.com/gitlab-org/gitlab/-/work_items/428236). 9. **Documentation and release post.** 10. **Rollout.** Define the feature flag strategy, enable on Dedicated, and validate the end-to-end flow. ## Open questions - Group-level certificates require the Enterprise User check. [gitlab-shell#841](https://gitlab.com/gitlab-org/gitlab-shell/-/work_items/841) argues instance-level does not need it, since the admin already controls all users. Does that reasoning hold for the DB-backed variant? * @vyaklushin I believe Enterprise User check can be skipped. Enterprise User is a GitLab.com feature. - Should certificate-based authentications, not just CA CRUD, also emit audit events? * @vyaklushin we can defer this decision to a follow-up. It's not a blocker for the implementation. - Should this use one feature flag for the whole feature, or separate flags for Rails and gitlab-sshd? How should it interact with the existing `FF_GITLAB_SHELL_SSH_CERTIFICATES` flag and the `ssh_certificates_rest_endpoints` history? * @vyaklushin Dedicated doesn't support feature flags. However, in this case `FF_GITLAB_SHELL_SSH_CERTIFICATES` is a ENV variable that can be controller by the deployment configuration. - Should there be an enforcement story at the instance level, such as requiring certificate auth or disabling personal SSH keys? Group-level has a similar issue in [#421917](https://gitlab.com/gitlab-org/gitlab/-/work_items/421917). This is likely out of scope for the first iteration. * @vyaklushin I agree it is out of scope. We can defer this one as well. --- Related: - [gitlab-shell#841](https://gitlab.com/gitlab-org/gitlab-shell/-/work_items/841) - file-based instance-level SSH certificates (shipped, insufficient for Dedicated) - Epic [&10662](https://gitlab.com/groups/gitlab-org/-/epics/10662) - group-level SSH certificates (reference implementation) - [Group SSH certificates API](https://gitlab.com/help/api/group_ssh_certificates) - [gitlab-sshd SSH certificates docs](https://docs.gitlab.com/administration/operations/gitlab_sshd_ssh_certificates/)