Do we really want to live stream the GCP migration?
I have some doubts about live-streaming our infrastructure migration tomorrow:
- Feedback from @yorickpeterse about last year's DB incident recovery which was live streamed
On Slack:
I noticed that we're live streaming the failover. Maybe I'm the only one, but can we not make that a thing for events like this?
I remember from last year's DB incident it was enough stress to deal with the issue, let alone with 5k people watching and going nuts in a chat
Even if you're not watching yourself, you know there are a lot of people watching you, judging your every move, etc
Last year it sort of made sense considering the impact of things. For stuff like this it honestly feels like you have to do your work in front of a firing squad except instead of bullets it's nasty tweets and comments
And in the company call agenda:
can we stop making live streams for this a thing? It’s stressful enough already, let alone having to do it front of thousands of grumpy users, ready to judge your every move. It doesn’t gain us anything, if anything I think it makes us look less professional by putting everybody involved out there (and this sentiment was shared by users during last year’s incident as well).
I think we may have dismissed @yorickpeterse's feedback very quickly, I think this is a very important feedback from a person that isn't even supposed to participate in the stream, and that had a really bad live stream experience in a similar situation before!
- Concerns were raised by @kathyw in today's GCP migration weekly call
Same here, I think we may have dismissed @kathyw's concerns quickly, I would say we should better be safe than sorry.
-
Concerns in today's GCP migration weekly call about the risk that our users/customers may think that this is risky / unprofessional to potentially leak their data in a live stream
-
From what I've seen we never publicly announced that we would live stream the migration: not in any blog post, only in one comment: https://about.gitlab.com/2018/07/19/gcp-move-update/#comment-3998849895.
-
IMO there's no added value to being transparent here, this is an internal / critical / risky migration, which is already very stressful in itself, live streaming it would just add more pressure to it. But based on the 2 reasons (and there are probably even more), I think we should really err on the side of safety and not live stream the migration.
-
If we think in terms of priority, the live stream is really a tiny detail compared to the migration itself and if there's any concerns about it (and I think there are some based on the above), I think we should not do it.
-
It seems to me we're committing to do the stream just because we internally said a long time ago that we would live stream the migration, but again, the live stream isn't the important thing tomorrow, and it's not too late to decide to not do it.
That being said, in the call, after some discussions, it was decided the following:
We’ll stream the nominal path, and set the expectation in the youtube video description that if things go off-nominal, we’ll shut down the live stream out of an abundance of caution so we don’t accidentally show an encryption key or customer data
and also in the company call agenda:
Comment noted and we are planning to retro after this weekend and this is a worthy point to evaluate with the data we get from this weekend on it. Update from GCP meeting is that we plan to livestream the happy path. If we run into issues, we plan to post that we are going to stop the livestream while we investigate any issues so that we don’t expose any secrets/keys/data accidentally.
But won't it be too late to discuss that after the fact?
WDYT @edjdev @kathyw @dawsmith @glopezfernandez @tommy.morgan @dhavens @meks @nick.thomas @ahmadsherif @alejandro @northrup @stanhu @digitalmoksha @mkozono @toon @ddavison