GitLab issueshttps://gitlab.com/gitlab-org/gitlab/-/issues2023-09-12T10:49:47Zhttps://gitlab.com/gitlab-org/gitlab/-/issues/7569Dynamic Secrets MVC2023-09-12T10:49:47ZMark PundsackDynamic Secrets MVC### Problem to solve
This MVC focuses on credential rotation / dynamic secrets / secret variables. Users need secure passwords/tokens, and an additional layer of security is to automatically rotate those credentials so that if any were ...### Problem to solve
This MVC focuses on credential rotation / dynamic secrets / secret variables. Users need secure passwords/tokens, and an additional layer of security is to automatically rotate those credentials so that if any were leaked, they'd be ineffective soon enough anyway.
From https://www.hashicorp.com/blog/why-we-need-dynamic-secrets
> Secret management is one of the core use cases for Vault. Today, many organizations have credentials hard coded in source code, littered throughout configuration files and configuration management tools, and stored in plaintext in version control, wikis, and shared volumes. Vault provides a central place to store these credentials, ensuring they are encrypted, access is audit logged, and exposed only to authorized clients.
>
> Achieving this centralization is a huge improvement in security posture, but its not the end of the journey. This is because applications don't keep secrets! It turns out, most applications do a worse job keeping secrets than our close friends. Applications frequently log configuration, leaving them in log files or centralized logging systems. Often secrets will be captured in exception tracebacks or crash reports sent to external monitoring systems, or they will be leaked via debugging endpoints and diagnostic pages after hitting an error. The list of ways applications leak secrets goes on, but the point is applications should not be treated as perfectly secure.
### Proposal
This MVC is at its core about adding credential rotation/dynamic secrets features to our bundled Vault.
Many users use Vault for this purpose, which we are planning on bundling with GitLab (https://gitlab.com/gitlab-org/gitlab-ce/issues/61548). Once this is available, we can build in automatic credential rotation into GitLab's variables (or allow a toggle via https://gitlab.com/gitlab-org/gitlab-ce/issues/40720). Using Vault also creates a smaller attack surface.
A good place to start with this automatic rotation is around Kubernetes cluster credentials, internal GitLab database credentials, but we should allow for any kind of secret to be managed in this way.
### What does success look like, and how can we measure that?
(If no way to measure success, link to an issue that will implement a way to measure this)
### Links / references
<!-- triage-serverless v3 PLEASE DO NOT REMOVE THIS SECTION -->
*This page may contain information related to upcoming products, features and functionality.
It is important to note that the information presented is for informational purposes only, so please do not rely on the information for purchasing or planning purposes.
Just like with all projects, the items mentioned on the page are subject to change or delay, and the development, release, and timing of any products, features, or functionality remain at the sole discretion of GitLab Inc.*
<!-- triage-serverless v3 PLEASE DO NOT REMOVE THIS SECTION -->Backloghttps://gitlab.com/gitlab-org/gitlab/-/issues/22815Review app composition for Auto DevOps2023-07-24T06:02:17ZMark PundsackReview app composition for Auto DevOps### Problem to solve
When using multiple services to generate an app, developers need an easy way to declare how to construct a review app.
### Further details
Developers can override the Helm chart to use, and that Helm chart can be ...### Problem to solve
When using multiple services to generate an app, developers need an easy way to declare how to construct a review app.
### Further details
Developers can override the Helm chart to use, and that Helm chart can be very complex, but there's currently only one Helm chart specifiable and it will be used for review apps, staging, and production.
### Proposal
Some ideas:
* Use environment-specific variables to specify the helm chart for different environments. This already works for Premium+.
* Detect multiple directories for Helm charts that are embedded in the app. Maybe `chart-test` or something like that.
* Detect `docker-compose-test.yml` and use that to generate a helm chart specifically for testing.
* More broadly, detect `docker-compose-<env>.yml` and `chart-<env>` so there can be different compositions for different environments. e.g. CI testing, review apps, staging, and production all use different configurations. Maybe using variables is better here since we already have environment-specific variables and inline charts (in the repo) are only supported for simpler cases. If you have multiple configurations, you have to use external repos and variables to point to them. Then again, environment-specific variables could point to directories in the repo, so maybe technically this would work anyway, at least for Helm charts. Maybe then we just need a variable to specify a docker-compose.yml (if/when we actually support Docker compose).
### What does success look like, and how can we measure that?
(If no way to measure success, link to an issue that will implement a way to measure this)
### Links / referencesBackloghttps://gitlab.com/gitlab-org/gitlab/-/issues/22466Reploy all environments after changing domain2023-07-24T06:02:15ZMark PundsackReploy all environments after changing domain### Description
If we let people set up Auto DevOps using a base domain based on `nip.io`, and they later add a real domain, do they need to redeploy all their projects again to get them working properly? Ingress will still work to the ...### Description
If we let people set up Auto DevOps using a base domain based on `nip.io`, and they later add a real domain, do they need to redeploy all their projects again to get them working properly? Ingress will still work to the old `nip.io` addresses, so it's technically OK, but won't they expect things to now be available under the new domain instantly?
### Proposal
* Reploy all (affected) environments whenever a base domain is changed.
### Links / referencesAwaiting further demandhttps://gitlab.com/gitlab-org/gitlab/-/issues/19675Automatically detect and enable services when deploying applications2023-07-24T06:02:14ZMark PundsackAutomatically detect and enable services when deploying applications### Description
Auto DevOps could be tuned to automatically provide different services. The next step is to allow auto detection of services reading project files (like Gemfile) and getting dependencies. We can, for example, consider th...### Description
Auto DevOps could be tuned to automatically provide different services. The next step is to allow auto detection of services reading project files (like Gemfile) and getting dependencies. We can, for example, consider that if there is the `pg` gem a PostgreSQL database should be provided by default. Lastly, some Heroku buildpacks do this automatically, and we could listen to the output of the `release` phase of buildpacks and automatically map any services requested to our own versions.
### Proposal
Automatically detect and enable services that are needed by the application, by defining the related variables (see https://gitlab.com/groups/gitlab-org/-/epics/127).
This should be done for the major languages (Ruby, Java, JavaScript, Go, etc).
<!-- triage-serverless v3 PLEASE DO NOT REMOVE THIS SECTION -->
*This page may contain information related to upcoming products, features and functionality.
It is important to note that the information presented is for informational purposes only, so please do not rely on the information for purchasing or planning purposes.
Just like with all projects, the items mentioned on the page are subject to change or delay, and the development, release, and timing of any products, features, or functionality remain at the sole discretion of GitLab Inc.*
<!-- triage-serverless v3 PLEASE DO NOT REMOVE THIS SECTION -->Backloghttps://gitlab.com/gitlab-org/gitlab/-/issues/6312Scale deployments directly from the deploy board2023-07-24T06:02:01ZMark PundsackScale deployments directly from the deploy board### Owners
UX: ?
FE: ?
BE: @matteeyah
### Description
Auto DevOps can leverage variables to define how many pods should be created during a deployment (`PRODUCTION_REPLICAS`, `CANARY_PRODUCTION_REPLICAS`, etc.): https://docs.gitlab.c...### Owners
UX: ?
FE: ?
BE: @matteeyah
### Description
Auto DevOps can leverage variables to define how many pods should be created during a deployment (`PRODUCTION_REPLICAS`, `CANARY_PRODUCTION_REPLICAS`, etc.): https://docs.gitlab.com/ee/topics/autodevops/#advanced-replica-variables-setup.
We should espose this variables in the **CI/CD > Environments** page, and allow users to easily change this values in the UI. After the change, we can automate the redeployment to apply the new number of requested pods.
### Original proposal
<details>
In the **CI/CD > Environments** view, for each deployment we can show
1. number of pods (e.g., `PRODUCTION_REPLICAS`)
1. if canaries are configured, number of canary pods (e.g., `CANARY_PRODUCTION_REPLICAS`)
1. if a deploy action is available in the related pipeline:
1. make it easy to change the values in the UI (since they are numbers, edit field with arrows to increase/decrease)
1. allow saving the new values, and automatically redeploy the application to apply the new values
This should support also environment-specific variables (related to https://gitlab.com/gitlab-org/gitlab-ce/issues/41436) in a transparent way for the user.
In the UI, we should make users aware that this flow works if they are using Auto DevOps, or if they implement a similar logic in their deployment jobs. It will link to documentation on how to leverage environment variables to deploy applications with a specific number of pods.
</details>
### Proposal
A new `Scale deployment` link is added next to the `Instances` title for each environment:
![scale-apps](https://gitlab.com/gitlab-org/gitlab/uploads/c92b6bed89479100f1498b344787b2bc/scale-apps.png)
When the link is clicked, the deploy board section is expanded inline to show the scaling options. The `Scale deployment` link is no longer available since we have entered scaling mode.
A message is shown at the top of the new section to inform users that this feature only works with certain configurations:
> Please, make sure your deployment job is compatible with scaling. Read [our documentation]() for more details.
TODO: Determine the documentation link
Below, the new option is shown. The description text informs the user that this change will affect deployments that happen after they save:
> Increase or decrease the number of instances used for this environment. This will affect future deployments.
The actual value is edited through a text field with arrow controls for easy modification.
Finally, there is a `Save` and `Cancel` button.
If the user collapses the environment, this section should be collapsed along with the rest of the deploy board.
![scale-apps--expanded](https://gitlab.com/gitlab-org/gitlab/uploads/943fbe6b6d8c1f14ace31189363ec64f/scale-apps--expanded.png)
We will check whether the configuration is compatible with deployment scaling:
>>>
1. Create `environment_scaling` that is gonna have all scaling options,
1. We gonna update `enviroment_scaling` once show the page,
1. We forbid updating `environment_scaling` if you have `PRODUCTION_REPLICAS` (for ex.) set as Secret Variable, as this is conflicting,
1. We show the error message, linking to edit of Secret Variables, the user deletes them, save, gets back to this page, clicks save and it's done,
1. We make `environment_scaling` to be included in variables sent to the runner.
>>>
If we detect that the configuration is incompatible, the default message is hidden and we show this message in a warning box:
> Some of the [secret variables]() in this project are incompatible with deployment scaling. Read [our documentation]() for more details.
[Secret variables]() will link to `project/settings/ci_cd`
TODO: Determine which docs page to link to for the second anchor
The field and `Save` button will be disabled.
The content of the field will be taken from the 'conflicting' variable.
![scale-apps--incompatible](https://gitlab.com/gitlab-org/gitlab/uploads/e8c5362bbc586514fde6efee53ce3a9b/scale-apps--incompatible.png)
<!-- triage-serverless v3 PLEASE DO NOT REMOVE THIS SECTION -->
*This page may contain information related to upcoming products, features and functionality.
It is important to note that the information presented is for informational purposes only, so please do not rely on the information for purchasing or planning purposes.
Just like with all projects, the items mentioned on the page are subject to change or delay, and the development, release, and timing of any products, features, or functionality remain at the sole discretion of GitLab Inc.*
<!-- triage-serverless v3 PLEASE DO NOT REMOVE THIS SECTION -->Awaiting further demand