Skip to content

Epic Work Items Rollout Plan

Conditions for Go vs No Go

Engineering

Product

Fallback plan and monitoring

Monitoring

What do we do in the case of mismatches after rollout?

  1. Evaluate with if it's an actual mismatch or false positive (check the prod database)
    1. Either you have read access to a replica, or you can use postgres.ai
    2. If there is replica access but you need to check data is required, you can request prod access via teleport
  2. Evaluate the impact (how many records are affected)
    1. The PLSQL script is a good starting point for the most common data we need to check and the amount of records affected
  3. Check with Product if the mismatch is a blocker for GA
    1. Depending on the data, product may decide that it's okay to still move forward with GA
  4. Evaluate if the mismatch comes from the WorkItem > Legacy Epic syncing or from Legacy Epic > WorkItem syncing
    1. When a mismatch comes from WorkItem to Legacy Epic syncing, it might be easier to rewrite the APIs to read from the WorkItem and get rid of the Legacy Epic data as SSoT.
  5. Evaluate if we need a full re-sync or if we could read from the legacy epic in the meantime
    1. To prevent a full re-sync of the data, we could also solve some mismatches by reading from the legacy epic.
    2. Use this approach with caution as it only works when the logic on the WorkItem side is correct.

How to rollout

Feature Flags

Feature Flag Description

work_item_epics

 Group-based feature flag

work_items_rolledup_dates

 Group-based feature flag

epic_and_work_item_associations_unification

Group-based feature flag.

⚠️️ This feature flag must not be rolled back once it has been rolled out to a group! Otherwise users would not see data they modified on the work item side.

Plan Team Member dogfooding

gitlab-org

  • Enable work_items_rolledup_dates for gitlab-org
  • Enable epic_and_work_item_associations_unification for gitlab-org
  • Enable work_item_epics for gitlab-org

gitlab-com

  • Enable work_items_rolledup_dates for gitlab-com
  • Enable epic_and_work_item_associations_unification for gitlab-com
  • Enable work_item_epics for gitlab-com

gitlab-data

  • Enable work_items_rolledup_dates for gitlab-data
  • Enable epic_and_work_item_associations_unification for gitlab-data
  • Enable work_item_epics for gitlab-data

GitLab Team Member dogfooding

  • Enable work_item_epics_rollout for gitlab team members. See docs for how to enable it on a feature group

    /chatops run feature set --feature-group=gitlab_team_members work_item_epics_rollout true

Before SaaS rollout

  • Get Change epic work items rollout per root-group (#478191 - closed) merged if we agree that we want to continue with SaaS rollout
    • Change work_item_epics feature flag to only check for the root ancestor
    • Enable work_items_rolledup_dates Observe the data as this changes the behaviour of dates calculation for groups that do not have work_item_epics enabled. If all is fine the flag can be removed
    • Enable epic_and_work_item_associations_unificationfor all and observe any potential performance problems. If all is fine the flag can be removed
  • Enable work_item_epics_rollout for all (This would enable everyone to see the epic work items on gitlab-org / gitlab-com). The flag can be removed too.

SaaS rollout

  1. Enable for all

    Since work_item_epics will enable work item epics per group with a percentage based rollout, we are not able to guarantee that the following feature flags are enabled for the same groups. For this reason we need to enable them for all.

    • epic_and_work_item_associations_unification
      1. ⚠️️ This feature flag must not be rolled back once it has been rolled out to a group! Otherwise users would not see data they modified on the work item side.
    • work_items_rolledup_dates

  2. Use a Percentage of actors rollout for work_item_epics

    Percentage rollout steps rollout steps:

    • 10%
    • 25% (after 12hs)
    • 50% (after 12hs)
    • 75% (after 12hs)
    • 100% (after 12hs)

Before SM rollout

SM rollout

  • Enable work_item_epics by default
  • Enable work_items_rolledup_dates by default
  • Enable epic_and_work_item_associations_unification by default

How to roll back

In the case if something goes wrong:

When only rolled out to plan team members

  • Disable work_item_epics on gitlab-org. It's fine to enable again on gitlab-org/plan-stage

When already rolled out to gitlab team members

  • Either disable work_item_epics for gitlab-org , or if we're fine to have plan-team members still with access, restrict feature flag actors on work_item_epics_rollout

When already rolled out to SaaS

  • Disable work_item_epics
  • Disable work_item_epics_rollout
  • Potentially: Disable work_items_rolledup_dates
    • We might not need to disable it. The cases when we should disable it are: syncing mismatches for dates, and syncing mismatches for the child hierarchy

Timeline

Date Description Notes
Aug 12 Finish pre-checks. Make sure all data is in sync

✅ Done
Aug 12

Plan-team dogfooding live 17.4

Enable epic work items on gitlab-org

✅
Aug 14

Plan-team dogfooding live

Enable epic work items on gitlab-com

✅
Aug 14 Pre-checks. Make sure all data is in sync

✅
Aug 14

GitLab Team member dogfooding live

Enable work_item_epics_rollout for feature-group gitlab_team_members

✅
Sept 4

Enable for all users of gitlab-org and gitlab-com

✅
Before Pre-checks. Make sure all data is in sync on SaaS

✅
October 10

SaaS go/no-go decision

✅
October 15

Enable on SaaS incrementally 17.6

✅
Before Pre-checks, Make sure all data is in sync for SM customers as reported by service ping

✅
TBC

SM go/no-go decision

✅
December 19

SM live with 17.7

✅
Edited by Amanda Rueda