Discuss when it's safe to halt an ongoing deployment
During #1641 (closed) and #1606 (closed), we've discussed the idea of halting an ongoing deployment so we can perform a rollback.
Canceling an ongoing deployment is not a common action and should be done carefully. Let's use this issue to discuss when it's safe to cancel a deployment and how we should perform such action.
Some questions that came up on the recent rollback demo:
- When a deployment is in progress, when it's safe time to cancel?
- When a deployment is in progress, how do we know it's safe to cancel?
- What happens to the nodes when we cancel a deployment?
- Who should cancel the deployment? RM, SRE on-call, both?
- Prior to canceling the deployment, should we drain canary?
- After canceling a deployment, what are the immediate actions? Can we immediately rollback?
🤖
Auto-Summary Discoto Usage
Points
Discussion points are declared by headings, list items, and single lines that start with the text (case-insensitive)
point:
. For example, the following are all valid points:
#### POINT: This is a point
* point: This is a point
+ Point: This is a point
- pOINT: This is a point
point: This is a **point**
Note that any markdown used in the point text will also be propagated into the topic summaries.
Topics
Topics can be stand-alone and contained within an issuable (epic, issue, MR), or can be inline.
Inline topics are defined by creating a new thread (discussion) where the first line of the first comment is a heading that starts with (case-insensitive)
topic:
. For example, the following are all valid topics:
# Topic: Inline discussion topic 1
## TOPIC: **{+A Green, bolded topic+}**
### tOpIc: Another topic
Quick Actions
Action Description /discuss sub-topic TITLE
Create an issue for a sub-topic. Does not work in epics /discuss link ISSUABLE-LINK
Link an issuable as a child of this discussion
Last updated by this job
-
⏫ ROOT Pipeline for code rollback &411 (closed) -
◀ PARENT Blocking a new deployment and attempting a rollback #1606 (comment 526011417) -
TOPIC When a deployment is in progress #1656 (comment 541393379)
- when it's safe time to cancel? #1656 (comment 541393379)
- how do we know it's safe to cancel? #1656 (comment 541393379)
- Production deployment will not happen without a RM present #1656 (comment 541690207)
- we can always stop at the beginning of the ansible pipelines #1656 (comment 542942657)
- we could stop between each service batch #1656 (comment 542942657)
- we must stop before running post-deployment migrations #1656 (comment 542942657)
-
TOPIC Feature improvement, release-tools can cancel a deployment #1656 (comment 541394220)
- I think this topic is a duplicate of #1627 #1656 (comment 542922895)
- TOPIC perhaps move away from mutable hosts? #1656 (comment 541880211)
Discoto Settings
---
summary:
max_items: -1
sort_by: created
sort_direction: ascending
See the settings schema for details.