Release Environment - Collect deployments data
Release Environment is a new platform and there are questions about it. We need to collect the data about its deployments and failures, with which we can answer some questions:
- How often a deployment to a release environment happen?
- How often does the deployments fail?
- For what reasons do the deployments fail?
- Are release environments reliable enough to be a part of the release process (i.e.
allow_failure: false
)? - Are smoke tests enough to detect issues with release environments? If not, what to add?
Exit Criteria
The following criteria based on what I think the information we need now. Feel free to add more.
-
Define the categories for deployment failures(e.g. QA, code change, Helm chart change, flaky external dependency, unreliable environment) -
List of all deployments after #19968 (closed) is done, with the following columns: - Time
- Target environment
- Failure reason (empty if success)
Collected Data
The collected data is gathered in a Google Sheet.
Result Summary
1. Successful/Failed Deployments
-
Total deployments (ignore testing): 33
-
Total failed deployments: 8.
Failure categories with the number of issues:
- QA: 4
- Code change: 0
- External Dependency: 3
- Release Environment: 1
-
Successful deployments (ignore testing): 25
Successful rate: 76%
2. Deployment Time
Note: The following numbers only count the successful deployments.
- Average Time of Release Environment Pipeline: 81 minutes
- Average Time of only Deployment step in Release Environment Pipeline: 5,6 minutes
- Mean Time of QA: 8 minutes
3. Other Conclusions
- When QA fails, it's stuck and timed out.
- This conclusion led to a reduce of timeout ( gitlab-org/gitlab!152381 (merged))
Edited by Dat Tang