Mark Delivery FY20Q2 OKRs
@gitlab-org/delivery @mayra-cabrera @nolith I've added provisional numbers for our FY20Q2 OKRs.
From each team member I expect to hear:
- GOOD - what went well
- BAD - what went poorly
- TRY - what we should try as a team and as individuals in the team.
For your convenience, copy the following into your comment and fill with details on whether you agree/disagree with the numbers (for the OKR's applicable to your work):
### Delivery FY20Q2 OKR results
* Streamline GitLab deployments on GitLab.com and self-managed releases => 100%
*
* Migrate GitLab.com registry to K8s and have it autodeployed as part of release process => 50%
*
* Complete GitLab codebase merge => 70%
*
* Implement storage limits => 50%
*
* Implement single object storage => 0%
*
### Delivery FY20Q2 Comments
* GOOD
1.
1.
* BAD
1.
1.
* TRY
1.
1.
Deadline for review is EOD 2019-07-25.
Merge request reports
Activity
added OKR label
- Resolved by Marin Jankovski
Delivery FY20Q2 OKR results
-
Streamline GitLab deployments on GitLab.com and self-managed releases => 100%
- Releases feel much more streamlined and we increased the number of deployments in all stages considerably so I think 100% is correct.
-
Migrate GitLab.com registry to K8s and have it autodeployed as part of release process => 50%
- I would score this a bit higher, 60%? We have registry running in kubernetes on staging, working on the readiness review for production and I don't expect a full quarter of work to get it over the line.
Delivery FY20Q2 Comments
- GOOD
- Reducing release toil be streamlining deployments
- Kubernetes demos were late in the quarter but what we have been doing so far has been helpful
- Considering the disruptive process changes that came with auto-deploy it went surprisingly well
- BAD
- We were very late to understand the importance of dog-fooding and we dithered a bit on decisions like using helm
- TRY
- Spend more time improving process and try sprint style retrospectives, I think this will help us manage priorities a bit better if we put more attention on unplanned work
- We spent more time on release auto-deploy related work than anticipated, perhaps we should be less aggressive with our goals?
Edited by John Jarvis -
- Resolved by Marin Jankovski
Delivery FY20Q2 OKR results
Streamline GitLab deployments on GitLab.com and self-managed releases => 100%
I think this went quite well, and we went from deploying maybe a few times per month to deploying a bunch of times per week (at least to staging and canary). I also liked the various handbook changes to clarify that work is not done until the code is running on GitLab.com, something we probably should have done much earlier.
Migrate GitLab.com registry to K8s and have it autodeployed as part of release process => 50%
I was not involved in this, so I can't comment on the matter.
Complete GitLab codebase merge => 70%
I think we made a lot of progress, though the goal of May 22nd was too optimistic. I recall we did have some issues early on with other teams not prioritising their share of the work, even though it had been made clear that this was high priority work.
One big challenge for me was the large number of unknowns. Since we are merging two big many-years-old projects, there are a lot of issues you may discover along the way. Some were easy to deal with, others took days of even weeks.
Implement storage limits => 50%
Not involved, so can't comment.
Implement single object storage => 0%
Not involved, so can't comment.
Delivery FY20Q2 Comments
- GOOD
- Single codebase work advanded nicely
- The auto deployer is working well so far
- BAD
- Not as much progress was made with the single codebase project as hoped for
- Not all teams prioritised the single codebase work as well as they should have (at least early on)
- GOOD
unassigned @yorickpeterse
unassigned @jarv
- Resolved by Marin Jankovski
Delivery FY20Q2 OKR results
- Implement storage limits => 50%
- I think here we were in a very uncomfortable spot. From the Engineering perspective, we found so much tech debt that we had to slow down and fix problems. Also this ended up as a multi-product-categories kind of feature, and many of the things to implement where Product/UX decision, but we have only backend engineers and a shared frontend engineer in our team
- Implement single object storage => 0%
- not exactly 0 maybe 1-5%. We started discussing it in https://gitlab.com/gitlab-org/gitlab-ce/issues/63097, we have a draft plan for JSON based uploads in gitlab-org/gitlab-workhorse#226 (moved) and I should complete this investigation by the end of the month (Q2 ends on July 31st)
Delivery FY20Q2 Comments
- GOOD
- we shed light on differences in storage limits across the codebase
- we addressed a DB performance issue
- BAD
- we ended up blocking each other because of the underlying tech debts
- TRY
- Maybe we need a PM for gitlab.com
- Maybe we need a PM for gitlab.com
edit: added my thoughts on Implement single object storage
Edited by Alessio Caiazza - Implement storage limits => 50%
- Resolved by Marin Jankovski
Delivery FY20Q2 OKR results
- Streamline GitLab deployments on GitLab.com and self-managed releases => 100%
- Was a security release part of this process? I'm not entirely sure this is a solid 100% due to such. Otherwise, yes. It's been great to see this change come to fruition.
- Migrate GitLab.com registry to K8s and have it autodeployed as part of release process => 50%
- I think this is fair. Our last demo showed a few remaining missing pieces that we still need to sort out, we still need to work on the migration strategy for production, and there are aspects of deployment methods that we've not yet discussed.
Delivery FY20Q2 Comments
- GOOD
- Our automation has improved tremendously for the release task process throughout the month. It's been a joy to see this come together.
- BAD
- The Kubernetes work is not happening as fast as I'd like, but we are making progress. I have a positive outlook that we can complete this work nicely in the next few months.
- Some aspects of auto-deploy have caused a few pain points with Product in determining how/when to advertise the completion of features.
- The ability to follow along with the auto-deploy/release process is tricky, and will be difficult to teach anyone new coming in. There's a lot of tooling in a variety of projects, many many pipelines and triggers to follow, and 3 total GitLab instances all involved to make things work properly, it's a tad mind boggling.
- TRY
- Provide feedback for Product. There are places where we utilize the product well, and other places where we could potentially abuse the product to force it to do something interesting that may be a benefit to others.
- May be planning better? Prime example is the Kubernetes work. Between moving quickly and running into road blocks late and creating something that we didn't fully think through, has lead us in directions that were not entirely anticipated or planned for and provides the appearance of us moving slower than we'd like.
- We should demo more often for every major project. This is a time to celebrate the success and provide a way to show off to others the improvements being made over time.
- Streamline GitLab deployments on GitLab.com and self-managed releases => 100%
unassigned @skarbek
- Resolved by Marin Jankovski
Delivery FY20Q2 OKR results
- Streamline GitLab deployments on GitLab.com and self-managed releases => 100%
-
Agree with Skarbek though, without a security process improvement I don't know that we're at 100% quite yet, but it's damn close.
-
- Complete GitLab codebase merge => 70%
-
we're pretty close.
-
Delivery FY20Q2 Comments
- GOOD
- Huge improvements to the release process. We've practically eliminated preparation MRs and cherry-picking (for pre-release), while deploying more often and with greater confidence.
- BAD
- Security releases are still delicate and require human toil.
- Broken
master
is still a huge issue, albeit one we're not directly responsible for. Many of these are due to CE changes breaking EE unexpectedly and will be resolved by single codebase.
- TRY
- In retrospect we punted some dog-fooding opportunities because the product wasn't viable for what we wanted yet, and wouldn't be in the timeframe we thought we were trying to hit. We could have moved slower up front in order to improve the product for everyone.
- Streamline GitLab deployments on GitLab.com and self-managed releases => 100%
- Resolved by Marin Jankovski
Delivery FY20Q2 OKR results
- Implement storage limits => 50%
- I think this number should be smaller, maybe 25%/30%?
- From my understanding, we manage to finish https://gitlab.com/gitlab-org/gitlab-ce/issues/59232, but there are several issues that we still need to do: gitlab-org&886, https://gitlab.com/gitlab-org/gitlab-ce/issues/59232#the-next-iteration
- Further progress is blocked until we finish the performance issue around updating namespace statistics (https://gitlab.com/gitlab-org/gitlab-ce/issues/62214). That issue is on the final phase, but still ongoing https://gitlab.com/gitlab-org/gitlab-ce/issues/64092.
Delivery FY20Q2 Comments
- GOOD
- We tackled a performance problem with a different paradigm: Updating things in the background instead of in the same transaction. https://gitlab.com/gitlab-org/gitlab-ce/issues/62214.
- Great teamwork to come up that solution (backend & database involved)
- Great teamwork to come up that solution (backend & database involved)
- We got rid of EE migrations in efforts of backporting the EE schema to CE https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/26940
- We tackled a performance problem with a different paradigm: Updating things in the background instead of in the same transaction. https://gitlab.com/gitlab-org/gitlab-ce/issues/62214.
- BAD
- For storage limits:
- When implementing storage limits, we encounter existent technical debt that leads us to more technical debt. https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/13163, https://gitlab.com/gitlab-org/gitlab-ee/issues/11675, https://gitlab.com/gitlab-org/gitlab-ce/issues/62214.
- We didn't have a PM or UX designer at the beginning which caused some confusion, and some extra meetings for Marin.
- For auto-deploy and single codebase (these are probably out of our control but worth mentioning):
- Besides auto-deploy was announced on multimodal communication (email, slack, engineering in review), engineers still rushed towards the feature freeze deadline (causing multiple broken master issues).
- Same thing happened with
db
migrations. After https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/26940, it was announced thatdb
migrations must be on CE, but some merge request slipped intomaster
with migrations onee
- For infrastructure backend engineers
- Backend engineers are new elements on the Infrastructure team, because of that, there's no onboarding process defined. That sometimes can slow down our participation in different projects, such as
ops.gitlab.net
, since first we need to request access (through slack), wait for the access to be granted and then open an issue/merge request.
- Policy access is being defined here https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/7205
- Backend engineers are new elements on the Infrastructure team, because of that, there's no onboarding process defined. That sometimes can slow down our participation in different projects, such as
- For storage limits:
- TRY
- Maybe Infrastructure people (
@nolith
and myself) can be more involved with Delivery tasks? Perhaps as backup Release Managers?
- Maybe Infrastructure people (
Edited by Mayra Cabrera - Implement storage limits => 50%
unassigned @mayra-cabrera
added my thoughts on Implement single object storage on !26287 (comment 194427880)
unassigned @nolith
unassigned @rspeicher
added 2442 commits
-
30fbed45...fe7cab70 - 2441 commits from branch
master
- 4bcf9a7b - Mark Delivery FY20Q2 OKRs
-
30fbed45...fe7cab70 - 2441 commits from branch
@glopezfernandez FYI.