For any new or existing system or a large feature set the questions in this guide help make the existing service more robust, and for new systems it helps prepare them and speed up the process of becoming fully production-ready.
Initially, this guide is likely to be used by Production Engineers who are embedded with other teams working on existing services and features. However, anyone working on a new service or feature set is encouraged to use this guide as well.
The goal of this guide is to help others understand how the new service or feature set may impact the rest of the (production) system; what steps need to be taken (besides deploying this new system) to ensure that it can be properly managed; and to understand what it will take to manage the reliability of the new system / feature / service beyond its' initial deployment.
## Summary
- [ ] Provide a high level summary of this new product feature. Explain how this change will benefit GitLab customers. Enumerate the customer use-cases.
- [ ] What metrics, including business metrics, should be monitored to ensure will this feature launch will be a success?
## Architecture
- [ ] Add architecture diagrams to this issue of feature components and how they interact with existing GitLab components. Include internal dependencies, ports, security policies, etc.
- [ ] Describe each component of the new feature and enumerate what it does to support customer use cases.
- [ ] For each component and dependency, what is the blast radius of failures? Is there anything in the feature design that will reduce this risk?
- [ ] If applicable, explain how this new feature will scale and any potential single points of failure in the design.
## Operational Risk Assessment
- [ ] What are the potential scalability or performance issues that may result with this change?
- [ ] List the external and internal dependencies to the application (ex: redis, postgres, etc) for this feature and how the it will be impacted by a failure of that dependency.
- [ ] Were there any features cut or compromises made to make the feature launch?
- [ ] List the top three operational risks when this feature goes live.
- [ ] What are a few operational concerns that will not be present at launch, but may be a concern later?
- [ ] Can the new product feature be safely rolled back once it is live, can it be disabled using a feature flag?
- [ ] Document every way the customer will interact with this new feature and how customers will be impacted by a failure of each interaction.
- [ ] As a thought experiment, think of worst-case failure scenarios for this product feature, how can the blast-radius of the failure be isolated?
## Database
- [ ] If we use a database, is the data structure verified and vetted by the database team?
- [ ] Do we have an approximate growth rate of the stored data (for capacity planning)?
- [ ] Can we age data and delete data of a certain age?
## Security
-[ ] Were the [gitlab security development guidelines](https://about.gitlab.com/security/#gitlab-development-guidelines) followed for this feature?
- [ ] If this feature requires new infrastructure, will it be updated regularly with OS updates?
- [ ] Has effort been made to obscure or elide sensitive customer data in logging?
- [ ] Is any potentially sensitive user-provided data persisted? If so is this data encrypted at rest?
## Performance
-[ ] Explain what validation was done following GitLab's [performance guidlines](https://docs.gitlab.com/ce/development/performance.html) please explain or link to the results below