# Summary This Epic tracks weekly status updates of the activities done by [Epic Work Items Migration Working Group](https://about.gitlab.com/handbook/engineering/development/dev/plan/working-groups/epic-work-items-migration/). ## Migration Proposal Keep this proposal updated with the current proposed steps for the migration <details> <summary>Technical migration steps (live) - Updated 2023-11-30 09:00 UTC</summary> The scope of this migration is to migrate epic records and its associations, but we don't plan to migrate epic boards ( https://gitlab.com/groups/gitlab-org/-/epics/12090#note_1664401846). 1. Exclude epic work items from all places issues appear (issue lists, APIs, RSS feeds, etc), except by enabling a FF. 2. Create an Epic Work Item Type (WIT) with all data attributes required to support Epic functionality. 3. Make sure no group level work items exists in the issues table (delete existing test work items) 4. When a legacy epic is created or deleted, create or delete a matching (namespace, iid) Epic Work Item (WI) (sync create/destroy). This sync should be done only if a feature flag is enabled for the group. 5. When a legacy epic is updated, find or create a matching Epic WI and update it(sync update). This sync should be done only if a feature flag is enabled for the group. 1. We can get the mechanism started with as little as iid and title just so we know each record has an equivalent on the work items table 6. Backfill the work items table with all existing legacy epics. We will insert the same IID that each record had in the epics table, in the issues table. https://gitlab.com/groups/gitlab-org/-/epics/12090#note_1661221698 7. At this point we should still prevent the creation of new work items at the group level. Only when we stop syncing from the legacy epics table we can allow new records to be created directly in the issues table without a match in the legacy epics table. This will be tricky when trying to develop epic work item features, but we could make it work in test groups that don't have any legacy epics. 1. If we would want to allow creation of other work items (other than epics) in a group for which legacy epics are synced, we would have to make sure that IID "counter" on issues table is synced back to epics table IID counter ( https://gitlab.com/groups/gitlab-org/-/epics/12090#note_1671736880) 8. Gradually migrate all attributes on legacy epic to **read** **from** the corresponding work item, instead of itself. 9. Gradually update all attributes on legacy epic to **write to** the corresponding work item, instead of itself, removing the sync update. 10. When all attributes are read/written from work item epic, all we need to keep on legacy epic record is `id` and `work_item_id` attributes. Before we allow users creating work item epics directly (through work items API), we should also make sure that when work item epic is created directly, we still create the thin legacy epic record (containing `id` and `work_item_id`), to assure that existing epic-related APIs still work as expected. https://gitlab.com/groups/gitlab-org/-/epics/12090#note_1671250036 1. this sync-back will allow us not having to replace legacy epic id references in DB 11. Roll out the FF and remove the legacy epic experience. Legacy APIs could write to and read from the issues table at this point if maintaining APIs is a requirement </details> ## Update Timeline **Note:** Updates are posted every Monday to include updates from the past week. <details> <summary> **2023-11-20** </summary> #### Confidence for FY25-Q1 Delivery - 70% <table> <tr> <th>Open tasks</th> <th>Comments</th> </tr> <tr> <td>Data syncing between Epics and Work Items table</td> <td> * As a part of splitting the effort towards this task, we have categorised multiple workstreams to identify DRIs. * There are 3 workstreams as of now; **Feature parity between Epics and Work Items, Syncing of epic changes to Work Items, and Backfilling epics to work items.** * Each workstream has a corresponding discussion thread in this epic, you can follow these threads for individual workstream status updates. * REST API compatibility after the data migration still remains a topic of concern so we discussed a number of approaches to solve it. We're also gathering usage stats of Epics Public API to identify which endpoints/queries need prioritization. * Frontend apps like Epics listing and Boards can potentially leverage Issues API with some modifications to include migrated epics since they all share the same Work Items table, however, we need to determine if tweaking Issues API is better or updating Epics API to switch data source (i.e. using Work Items table) is ideal. </td> </tr> </table> </details> <details> <summary> **2023-11-13** </summary> #### Confidence for FY25-Q1 Delivery - 70% <table> <tr> <th>Open tasks</th> <th>Comments</th> </tr> <tr> <td>Data syncing between Epics and Work Items table</td> <td> * Team had a detailed discussion during the Weekly WG Sync as well as a focused technical discussion call between @johnhope, @acroitor & @jprovaznik ([recording](https://drive.google.com/file/d/1CIlT_sZMUgOq\_-wLW0mvg-bnGY8d-eFh/view?usp=sharing) available to view internally). * There were two options to choose from when it comes to sync strategy; Unidirectional Sync and Point-in-time Migration. * **Based on weighing the pros and cons of both approaches, the team has decided to go with the _Unidirectional Sync Strategy._** * With Unidirectional syncing, the next step is to determine a plan for making Epic frontend apps, specifically Boards and Roadmap, compatible with Work Items API, this is being discussed in [Slack thread (internal)](https://gitlab.slack.com/archives/C02403NRD40/p1699573794132989) as well as in &amp;9292. * Given that the strategy is now finalised, we'll also be working towards identifying DRIs for each task of the migration preparation. </td> </tr> <tr> <td> [Work item - Epic Start/Due Dates](https://gitlab.com/gitlab-org/gitlab/-/issues/425201 "Draft: Spike: Work items - Epic Start/Due Dates") </td> <td> * @kassio (in collaboration with @egrieff) made more progress towards inheriting dates widget for Epic Work Items !134921, including a demo! </td> </tr> </table> </details> <details> <summary> **2023-11-06** </summary> #### Confidence for FY25-Q1 Delivery - 65% <table> <tr> <th>Open tasks</th> <th>Comments</th> </tr> <tr> <td> [Work item - Epic Start/Due Dates](https://gitlab.com/gitlab-org/gitlab/-/issues/425201 "Draft: Spike: Work items - Epic Start/Due Dates") </td> <td> * @kassio has opened https://gitlab.com/gitlab-org/gitlab/-/merge_requests/134921 as PoC towards roll-up of start and due dates from child work item milestone. * @donaldcook gathered data on Epics within GitLab.com and distribution of epics with fixed dates vs. inherited dates https://gitlab.com/gitlab-org/gitlab/-/issues/424544#note_1629238970. This allows us to make a decision on whether we want to support individual inherited dates or a single `Inherit` option that governs both the start and due dates together. </td> </tr> <tr> <td>Data syncing between Epics and Work Items table</td> <td> * The team spent more time this week discussing whether data syncing is indeed a good approach as we're looking at potentially indefinite two-way syncing of tables which is costly. * An argument in favour of data syncing is to support legacy Epics API, particularly REST API, and with the lack of REST API support in Work Items, there's no appropriate replacement available. * On the other hand, two-way data syncing looks simple for basic widgets like Labels, Dates and Ancestry, but is complex for Hierarchy and Linked Work Items widgets where we need to maintain relationship and permissions model in sync between both the tables. * We're exploring alternative approaches in https://gitlab.com/gitlab-org/gitlab/-/issues/430242, one of the approaches presented is [Unidirectional Sync](https://gitlab.com/gitlab-org/gitlab/-/issues/430242#note_1632206189 "Spike: Migration Plan for legacy Epics to Epic Work Item Type"). </td> </tr> </table> </details> <details> <summary> **2023-10-30** </summary> #### Confidence for FY25-Q1 Delivery - 65% <table> <tr> <th>Open tasks</th> <th>Comments</th> </tr> <tr> <td> [Work items - Epic Start/Due Dates](https://gitlab.com/gitlab-org/gitlab/-/issues/425201 "Draft: Spike: Work items - Epic Start/Due Dates") </td> <td> * While the confusion around Inherit vs. Roll-up remains especially for epics, we need to aim for parity between epics and work items for now. * We'd eventually get to address the terminology to resolve this confusion in the long term. * Adding support for inheritance on the current Start/Due date widget while selectively disabling it for certain work item types will make implementation complex. * We have determined that creating a separate _date range_ widget just for Epic Work Item type is more appropriate as then data syncing will be easier. * In situations where this date-range widget is used, we need to make the start/due date widget mutually exclusive of this inherit-supported date-range widget for any work item type. </td> </tr> <tr> <td> [Relationship widget rollups](https://gitlab.com/groups/gitlab-org/-/epics/11402 "Hierarchy widget rollups") </td> <td> * We discussed the pros and cons of keeping roll-up data (i.e. counts of children per type, sum of weights, etc.) on the hierarchy widget vs. building a separate read-only widget that contains these counts. * Having a separate widget just for roll-ups allows for plugging that information in places beyond the hierarchy widget, this also allows for managing certain statistics easily in cases where separation of user weight vs. roll-up weight is needed. </td> </tr> <tr> <td>Data syncing between Epics and Work Items table</td> <td> * @felipe_artur opened a small PoC MR [!134920](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/134920 "Draft: Sync epics as work item") to check what it takes to setup syncing. Feel free to ask questions or provide feedback in the MR. * The approach we've thought of is to start with syncing core data of epics (i.e. title, description, confidentiality) and then sync widget data as widgets become ready. </td> </tr> <tr> <td>Documentation on widget availability per work item type</td> <td> * @amandarueda has [created a support table](https://gitlab.com/groups/gitlab-org/-/epics/6033#widget-availability-by-work-item-type "[INITIATIVE] Work Items") to list down which work item widgets are available on which types. * We've opened [!135353](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/135353 "Add Work Item widget availability table") to add this table to [Work Items Architecture Blueprint](https://docs.gitlab.com/ee/architecture/blueprints/work_items/) document. </td> </tr> </table> </details> <details> <summary> **2023-10-23** </summary> - Discussed if syncing Epics and Work Items (i.e. Issues) table an only option or we can look at actual data migration. - The biggest blocker in doing data migration and dropping the Epics table is the lack of REST API support for Work Items. While we've made [some progress](https://gitlab.com/gitlab-org/gitlab/-/issues/368055 "Add Work Items to the REST API") in exploring the possibility of building REST API for Work Items but the work is far from over. - Discussed concern around change in Epic IDs once they're copied over to Work Items table as accessed via Work Items API. - One of the possible approach is to continue allowing to fetch epic data via older reference but still something we'd have to look into once we actually start data copying. - Discussed blocker for DB syncing of Epics table and Work Items table in first sync call. - We identified that besides [Epic Color](https://docs.gitlab.com/ee/user/group/epics/manage_epics.html#epic-color) and [Start and due date inheritance](https://docs.gitlab.com/ee/user/group/epics/manage_epics.html#start-and-due-date-inheritance), work items support everything that Epics support, so to accelerate our db syncing plan, we've taken following steps; - We have determined that Color support is already behind FF and we haven't defaulted it so we can retire it during the migration - For dates inheritance, we've decided to continue the same behaviour as current (i.e. milestone assigned to child issue influences dates of parent epic), so we have scheduled https://gitlab.com/gitlab-org/gitlab/-/issues/425201 for %"16.6" and @kassio & @himkp will be helping with the same. </details> <details> <summary> **2023-10-16** </summary> - WG page created [here](https://about.gitlab.com/handbook/engineering/development/dev/plan/working-groups/epic-work-items-migration/) - First weekly sync scheduled for 2023-10-18 to discuss immediate actions required to set data sync between the Epics table and Work Items table </details> --- ### Feature Flags <table> <tr> <th>Flag</th> <th>Description</th> <th>Status</th> <th>Notes</th> </tr> <tr> <td> `work_items_mvc_2` </td> <td> Alpha-level work item features (enabled on `gitlab-org/plan-stage`) </td> <td> **default enabled:** false **staging:** enabled globally **.com:** * `gitlab-org/plan-stage` * a couple projects </td> <td> </td> </tr> <tr> <td> `okrs_mvc` </td> <td> OKRs planning inside GitLab (enabled on `gitlab-com/gitlab-OKRs`) </td> <td> **default enabled:** false **staging:** not enabled globally **.com:** a group and some projects </td> <td> </td> </tr> <tr> <td> `namespace_level_work_items` </td> <td>Work items on the group-level</td> <td> **default enabled:** false **staging:** enabled globally .com: a few groups </td> <td> </td> </tr> <tr> <td> `work_item_epics` </td> <td>Enables epics to be rendered as epics per group</td> <td> </td> <td> </td> </tr> <tr> <td> `work_item_epics_rollout` </td> <td>Enables access to epic work items per user</td> <td> **default enabled:** false **staging:** about to be enabled when deployed </td> <td> </td> </tr> <tr> <td> `validate_epic_work_item_sync` </td> <td>Runs diffing between epic and the synced work item on update or create to find mismatches</td> <td> **default enabled:** false **staging:** enabled globally **.com:** enabled globally </td> <td> </td> </tr> <tr> <td> `work_items_rolledup_dates` </td> <td> Calculates the start/due dates within a hierarchy for work items. Also used on Legacy Epics to use the WorkItems roll up logic to calculate the dates </td> <td> **default enabled:** true **staging:** enabled globally **.com:** enabled globally </td> <td> </td> </tr> <tr> <td> `epic_and_work_item_associations_unification` </td> <td>to enable delegation of other associations (e.g. subscription, labels, notes)</td> <td> default enabled on staging. .com: `plan-stage` </td> <td> </td> </tr> </table> ### Useful Links - [Weekly Sync Recordings](https://www.youtube.com/embed/videoseries?list=PL05JrBw4t0KrEtDK8xRruSY5VtQb9DmdX) - [Work Items Overview (Slides)](https://docs.google.com/presentation/d/1uvbcCMh0WBWp7QIQhttCiy849PuD-dJO6Mf_OoXP_nA/edit) - [Work Items Contribution Guidelines](https://docs.gitlab.com/ee/development/work_items.html) - [Architecture Blueprint](https://docs.gitlab.com/ee/architecture/blueprints/work_items/) ### #####