Skip to content

[Discussion] Custom Fields async discussion

Context

  • We will soon add support for several different "configurable" aspects of the work item architecture (e.g., custom fields, and configurable types).
  • Our knowledge architecture already has several constraints that make it difficult to properly support common planning use cases (see Problems with GitLab's Knowledge Architecture (#467558)).
  • The inheritance behavior (cascading down, rolling up) across the plan surface area is very inconsistent.
  • We must provide customers with mechanisms to standardize configuration optionally while providing individual teams the autonomy to optimize GitLab for their specific use cases.
  • Organizations (part of Cells) are not built on top of Namespaces, and there is little appetite for integrating Namespaces with Organizations. Organizations will also not be rolled out to customers until mid-to-late CY25. Integrating any plan capabilities with Organizations in the future will be problematic.

Upcoming features for devopsplan

Guiding principles and goals

  1. Provide teams the flexibility to reflect their processes in GitLab.
  2. Empower customers to standardize configuration across groups/projects to ensure consistent outcomes.
  3. Minimize complexity so that any user is able to understand and manage configuration.
  4. No performance hit.
  5. Avoid decreasing flow/cycle time.
  6. Enable a smooth transition from the current label-based model.

Current inheritance behavior

Object Visible in parent namespace object list Filterable in parent work item list List option in parent board Filterable in parent board Visible in child namespace object list Filterable in child work item list List option in child board Filterable in child board Child work items can be associated to parent namespace object Parent work items can be associated to children namespace object
Issue ✅ ✅ ✅ ✅ N/A ❌ N/A ❌ ❌ N/A
Epic ✅ ✅ ✅ ✅ ❌ ✅ N/A ✅ ✅ ✅
Milestone ✅ ❌ ❌ ✅ ❌ ✅ ✅ ✅ ✅ ❌
Label ❌ ✅ ❌ ❌ ✅ ✅ ✅ ✅ ✅ ❌
Iteration ❌ ❌ ❌ ❌ ✅ ✅ ✅ ✅ ✅ ❌

Scenarios

Given the following:

flowchart TD
  Organization --> Group_A --> Subgroup_Y --> Project_1
  Organization --> Group_B --> Subgroup_X --> Project_2
  Group_B --> Work_Item_Types_B --> Feature
  Group_B --> Custom_Fields_B --> B_Text["Department Code (select input)"]
  B_Text .-> Feature
  Subgroup_X --> Work_Item_Types_X --> Bug
  Subgroup_X --> Custom_Fields_X --> X_Int["RICE score (integer)"]
  X_Int .-> Bug
  Group_B --> Project_3
  Project_3 --> Work_Item_Types_3 --> Chore
  Project_3 --> Custom_Fields_3 --> 3_SELECT["Reviewer (select list)"]
  3_SELECT .-> Chore

Current HIGH Risks

  • Usability:
    • Will customers intuitively understand how the upcoming configuration-related features behave within the context of our group/project hierarchy?
    • Will our approach provide adequate controls for standardization if desired, while balancing the desire for some customers to provide more autonomy within each team?
    • Will our decisions now scale to when we tackle solving "horizontal workflows" (ex: Issue from Group A is visible somewhere in Group B -- #467558)
  • Performance:
    • Aggregation views (boards, lists, ...), especially groups, are already strained by the current approach of traversing all children namespaces to check permissions. Will our approach be scalable and performant?
    • Is there another cascading/rollup pattern we should explore that will scale better AND provide a performance improvement to our current aggregation view experience?
    • What are the potential performance and complexity trade-offs of the different options? Which would be ideal from an engineering/architecture standpoint?

Previous art

Status Updates

  • 2024-08-14 Sync
    • Recording: https://youtu.be/gfUUUFfkPNQ (must be logged in to unfiltered)
    • TL;DR: Discussed long-term vision and backed into MVC. Heinrich is going to spend the next day putting together a rough implementation plan based on an MVC that will add basic custom fields (select list, text input) to root namespaces, associating a field with a work item type using a policy model where the policy would cascade down a namespace hierarchy.
  • 2024-08-15 Sync
  • 2024-08-16 Sync
    • Recording: https://youtu.be/DnC8szNZcpk
    • TL;DR: Reviewed work breakdown for MVC1 (&235), made a few more decisions (documented below). Next Step: Convert work breakdown into issues and tasks. Engineering is confident in delivering MVC1 (at least the backend portion). We need FE engineering to weigh in with their thoughts. We can work on custom fields in parallel to getting issues migrated to work items. We can release custom fields for Epics, OKRs, and Tasks as soon as MVC1 is ready or wait until issue migration has taken place. We can also potentially render the work item custom field widget on legacy issues, but that would add a bit of extra throw away work and we would not want to add the custom fields widget to the legacy issue creation experience. UX and Engineering can weigh in on this next week if it's a scenario we want to explore further.

Decisions Made

  • Schema: Evaluated the pros and cons of an EAV vs. JSON schema for custom fields. The decision was made to proceed with the EAV model because it will handle referential integrity, sorting, and other things we will want to do better than JSON.
  • Custom work item types: With the proposed schema, this is not a dependency. For example, customers will be able to add custom fields to our existing default work item types without having to create a custom type first.
  • Extensibility: Custom fields will be decoupled from the work item widget schema. Work items will have one "custom field" widget, which contains all of the custom fields configured for that given type. This also means that:
    • Subnamespaces can eventually have the ability to create custom fields and associate them to work item types inherited from the root group.
    • Custom fields will be able to work with other objects in GitLab, including custom objects, if we opt to prioritize this in addition to configurable work item types in the future.
  • MVC Scope:
    • Limit configuring custom fields to root namespaces
    • Support the ability to map custom fields to default work item types
    • Initial custom field types: Select field, number, text input
    • Filtering: Yes, but only for field types that make sense (e.g. select field type would allow filtering based on option value, but for text inputs, there is nothing to filter on)
    • Cascading: Custom fields and field <> work item type mappings will cascade to children namespaces. These will be visible in child namespace work item configuration screens, but will only be editable if you have the appropriate permissions in the root namespace.
    • Updating a custom field on a work item will add a system note on that work item.
    • Moving:
      • Moving a single work item (or bulk move) to a different namespace hierarchy will remove all custom fields from the current namespace hierarchy. In the UI, handle as a destructive action
      • Moving a subnamespace to a different namespace hierarchy will remove all custom fields and values from all work items in that namespace and below. In the UI, handle as a destructive action
    • Emphasize in the UI: Changes made to existing custom fields (e.g. Renaming a select field's option name, deleting an option, deleting a custom field) is a potentially destructive action for all work item types that use the custom field and have values already defined. Show a double confirmation for destructive actions, including the count of work items that the change would impact.
    • Changing types: The same behavior as moving, including warnings about destructive side effects of potential data loss.
    • Archiving or deleting custom fields, including warnings about descrutive side effects of data loss (not applicable if we opt for archiving).
    • Notifications and webhooks (stretch)
  • Naming collisions: We will not worry about potential naming collisions between custom fields and future system default widgets. Customers can work around this by changing the name of their custom fields if there is confusion. We can defer work on the policy concept until we introduce the ability for children namespaces to configure their own custom fields (or until we get to status lifecycles first).
  • CSV Import/Export: For CSV Import/Export, custom fields and values must exist within a namespace before import. In the event of a naming collision between a custom field and a system field, the system field will take precedence over the custom field. Import/Export will be deferred until MVC2.
  • Archiving vs Deleting: There is not much difference in effort between deleting and archiving fields. Archiving would preserve the field values on all work items but would not be visible anywhere in the UI. Unarchiving a field would restore the original values at the time it was archived. UX can weigh in on this decision next week.
  • Configuration Pathways: If we are in a time crunch, we will only support one of the two pathways for creating and configuring a custom field for MVC1.
  • Field Priorities: The implementation priorites for field types will be Select, Number, and String. This will prevent customers from inadvertently storing numbers in string fields that they may want to use when we introduce calculated fields. We will focus on the end-to-end workflows across all views for the select field type before prioritizing the number and string types.
  • Notifications & Webhook Events: Notifications and webhook events are included in MVC1 as a stretch goal. This will be prioritized as one of the last items we work on for MVC1. If desired, we can decide to shift this to MVC2 at a later date.
  • Quick Actions: Quick actions will not be included in MVC1. In the interim, we need to collaborate on the appropriate syntax for custom field quick actions that is extensible to all field types.
  • Re-ordering Fields: We will not allow re-order custom fields and/or system default widgets in the sidebar at this time.
  • Custom Fields Application Limits: Namespaces will be limited to 30 total custom fields available for assignment to all work item types. A Work Item Type will be limited to a maximum of 10 custom fields per type. These values were arbitrarily decided and are subject to change before the release of MVC1, or after the release depending on feedback and the impact custom fields have on system performance, availability, or perceived responsiveness of the application.

Follow-up questions on schema and MVC scope definition

  • Are we still planning on leveraging the idea of "policies" to group configuration options for work item type < > field mapping, collections of default and custom types, and configurable statuses? Is this something we need to lay the foundation for in the initial MVC?
    • We can still use policies. We may not need to with EAV given we can add custom fields in child namespaces to types inherited from parents and hide custom fields inherited from parents (if allowed)
  • How does the custom field schema work when we get closer to prioritizing custom objects? Would those objects essentially be a different solution with a foreign key in a custom field?
    • If we implement custom objects as work item types, then no change is needed.
    • But if they’re a separate object, we can re-use the same custom field configuration. But then have a new table to map custom fields to custom objects. Similar to the table we have to map custom fields to work item types.
  • Will it be possible to extend custom fields to work with other objects in GitLab (e.g. Project, User, Merge Request, ...)
    • Yes
  • How do we want to handle the potential conflict in naming for future system default fields for custom fields (e.g. as soon as we release custom fields, customers will create a "Priority" custom field. In the future, we want to solve the prioritization use cases in a first-class way). If we want to create a Priority widget and there are already custom fields named Priority, this could be confusing.Should we do anything now to help head off potential naming conflicts in the future?
    • It wouldn't cause any data conflicts, but it would make the lives of customers more painful. They can easily go in and change the name of a field.
  • For CSV import/export, how do we want to handle custom fields? Suggestion - custom fields must be created before import, only import matching custom fields. Export all work item data.
    • We can do this, but engineering recommends doing this in MVC 2. The import/export needs a lot of love to even support existing default widgets. If we have time in Q3. Duplicate widget names maybe be problematic but we can have a precedence order for preferring system default widgets.
  • How much extra work would it be to support multi-select along with single-select?
    • Shouldn't be much work. This would be a nice to have for MVC, but not required.

Decision Outstanding

For the first MVC:

  • UX - Do we need to show custom fields on work items in aggregation views (e.g. Issue list rows, epic list rows, board cards, ...)
    • Gabe votes no. (insert link to Product Planning's issue for display control enhancements)
  • UX~~ - Do we need to support the ability to re-order custom fields and/or re-order widgets in the sidebar?~~
    • Gabe votes no for the MVC (and maybe ever)
  • UX~~ - UI structure for mapping custom fields to work item types:~~ - Option A: Navigate to work item type > add new or existing custom field - Option B: Navigate to custom fields section > add new or edit custom field > select applicable work item types
    • Option C: Both Option A or Option B. You can navigate to the custom fields area to manage your field library and control field-to-work item type mapping OR add new fields to your field library and/or add an existing field from within the work item type configuration view.
  • UX - Do we need to support sorting in aggregation views based on custom field values (e.g. Given a "Priority" select custom field type with values of 1, 2, 3, 4 where the value order is defined based on the position in the custom field configuration could be sorted within an issue list by the field's value order.)
    • Gabe votes no for the MVC. Group with display control enhancements planned for later in Q3.
  • UX~~ - Do we need to support multi-select along with single-select for the MVC?~~
    • Gabe votes no, unless there is not much extra engineering effort.
  • UX~~ - Do we need to support custom field updates triggering notifications and webhook events?~~
    • Gabe votes yes
  • UX - Do we need to support bulk editing custom fields?
    • Gabe votes no until we rework the bulk editing experience
  • UX & BE~~ - Should we do archiving of custom instead of/before deleting to be less destructive? I wish we did this for labels, milestones, and iterations 🤔~~
    • Gabe votes maybe depending on engineering and UX feedback
    • Archiving would essentially hide the custom field from the custom field widget
  • UX~~ - Import/export?~~
    • Gabe votes maybe depending on effort?
  • UX & BE~~ - Support for quick actions in the MVC? What would this look like?~~
    • /field :name (autocomplete) :values (autocomplete for select list)

For follow-up MVCs:

  • For custom fields owned by subnamespaces, do we show them as filter options and in work item aggregation views in parent namespaces?
    • Gabe thinks this is a bad idea.
  • Do we want to allow the creation of board lists based on applicable custom field type values?
  • ...there are a lot of more we'll continue to add as discussions move forward.
Edited by Gabe Weaver