## 1. Problem Statement We need to implement fine-grained visibility controls for work item types across namespace hierarchies. Without this implementation type availability is binary—a type is either available everywhere in a hierarchy or nowhere. We want to support three independent visibility actions per namespace and type: 1. **Enable/Disable for THIS namespace only** - Control visibility for the current namespace without affecting children 2. **Enable/Disable for ALL existing children** - Propagate visibility to all descendants (overrides other distinct settings on child namespaces) 3. **Enable/Disable for NEW children only** - Set default visibility for future namespaces created under this namespace These three actions are independent. For example, you can enable a type for this namespace, disable it for all existing children, but enable it for new children created in the future. **Key constraints:** - Must work with infinitely nested namespace hierarchies - Must handle several thousand namespaces per tenant - Maximum of 40 types per root namespace - Resolution happens in bulk (all types at once) at request start via the `WorkItems::TypesFramework::Provider` abstraction layer - Default state is `enabled` unless explicitly overridden ## 2. Why the Obvious Approaches Don't Work ### Approach A: Pre-materialize Everything Store a row for every namespace × type combination (potentially thousands × 40 = hundreds of thousands of rows). **Why this fails:** - Massive storage bloat - most namespaces inherit defaults and don't need explicit rows - Index bloat degrades all queries - Vacuum overhead increases - Most stored data is redundant (inherited defaults) ### Approach B: Resolve at Runtime with Complex Queries Store nothing, compute visibility by walking the entire hierarchy on every request. **Why this fails:** - Slow reads - we need to resolve visibility for 40 types at request start - Complex recursive CTEs for every request - Cannot leverage indexes effectively when traversing arbitrary depths - Read performance degrades with hierarchy depth Both extremes optimize one axis at the severe expense of the other. We need a middle ground. ## 3. Our Constraints We need an architecture that satisfies: 1. **Fast reads** - Ideally a single PostgreSQL query to resolve all 40 types 2. **Acceptable writes** - Mutations can be async for bulk operations, but should be reasonably fast 3. **Minimal storage** - Store only explicit overrides, not inherited defaults 4. **All three actions supported** - Each action (THIS, ALL_CHILDREN, NEW_CHILDREN) must work efficiently 5. **Namespace moves are cheap** - Moving a namespace in the hierarchy shouldn't require rewriting visibility data 6. **Default is enabled** - If no override exists, types are visible ## 4. Global Visibility Toggle Before implementing fine-grained controls, we need a top-level toggle to enable/disable the entire visibility system. I propose we'll introduce a `work_item_settings` table: ```ruby work_item_settings id organization_id (bigint, nullable, indexed, FK → organizations) namespace_id (bigint, nullable, indexed, FK → namespaces) customizable_type_visibility (boolean, default: false) created_at updated_at # Ensure exactly one container is set constraint: check ((organization_id IS NOT NULL)::int + (namespace_id IS NOT NULL)::int = 1) ``` **Why dual columns?** For SaaS deployments, we use the top-level namespace as the container for custom types. For self-managed installations, we use the `Organization` model. Both columns allow us to support both deployment models during the migration period. Long-term, we'll migrate everything to organization-level, at which point we can drop the `namespace_id` column. Because of this we cannot reuse the existing `namespace_settings` table and similar concepts. **Future extensibility:** This table is designed to hold additional work-item-related settings in the future. For example, we're planning to add a `default_work_item_type_id` column to specify which type should be used as the default for various actions (we use `Issue` today, but there'll be a world where `Issue` might not exist). This keeps all work-item configuration in one place. **Resolution flow:** 1. Check `work_item_settings.customizable_type_visibility` for the container 2. If `false` → return all non-archived types (current behavior) 3. If `true` → apply the visibility resolution logic described below We should be able to preload `work_item_settings` via a scope on `Organization` and `Namespace` models on the resolver level so we can expose that setting via GraphQL. We can either take the argument on the `organization` or `namespace` level or use a separate mutation for this. ## 5. GraphQL Interface ### Mutation We'll [expose a single mutation](https://gitlab.com/groups/gitlab-org/-/epics/19883#note_3071749459): ```graphql mutation { workItemAvailabilityToggle( input: { namespaceId: "gid://gitlab/Namespace/123" workItemTypeId: "gid://gitlab/WorkItems::Type/5" # or custom GID action: ENABLE # or DISABLE scope: THIS # or ALL_CHILDREN or NEW_CHILDREN } ) { errors } } ``` **Action values:** - `ENABLE` - Make the type visible - `DISABLE` - Make the type hidden **Scope values:** - `THIS` - Apply only to the specified namespace - `ALL_CHILDREN` - Apply to all existing descendant namespaces (destructive, async allowed) - `NEW_CHILDREN` - Apply to future namespaces created under this namespace ### Query When fetching work item types, we already expose an `enabled` boolean flag. Currently, this is controlled by: - Group namespaces: Only epics are enabled - Project namespaces: All types except epics (custom types limited to project level for now) We fetch available types in `WorkItems::TypesFramework::Provider`, which: 1. Merges available custom and system-defined types 2. Stores them in `RequestStore` for reuse during the request in a lookup table We'll add a visibility resolution layer that: 1. Fetches visibility settings for the current namespace (single query for all 40 types) 2. Sets `enabled` flag based on resolved visibility 3. Sets the type objects in `RequestStore` This integration point is clean — we're adding one additional resolution step before caching. ## 6. Architecture: Sparse Inheritance with Ancestor Resolution (SIWAR) We'll implement what we're calling **Sparse Inheritance with Ancestor Resolution (SIWAR)**. The core idea: store only explicit visibility decisions (overrides) and resolve inheritance at read time using the `traversal_ids` array in a single query for all available types. ### Database Schema ```ruby work_item_type_visibilities id namespace_id (bigint, indexed, FK → namespaces) work_item_type_id (bigint, indexed, FK → work_item_types) enabled (boolean, not null) propagate (boolean, not null, default: false) created_at updated_at unique_index: [namespace_id, work_item_type_id] work_item_type_visibility_defaults id namespace_id (bigint, indexed, FK → namespaces) work_item_type_id (bigint, indexed, FK → work_item_types) enabled (boolean, not null) created_at updated_at unique_index: [namespace_id, work_item_type_id] ``` **Column explanations:** - `enabled` - Whether the type is visible or hidden - `propagate` - Whether this override should apply to descendant namespaces (used during resolution) ### How It Works **Sparse storage:** We only store rows for namespaces that have explicit overrides. If a namespace inherits default behavior, no row exists. **Ancestor resolution:** When resolving visibility for a namespace, we query all ancestors (via `traversal_ids`) and find the closest ancestor with an override. **Future rules:** The `work_item_type_visibility_defaults` table stores rules that apply to namespaces _created in the future_ under a given namespace. ### Resolution Query ```sql -- For namespace with traversal_ids = [1, 5, 12, 45] and work_item_type_ids = [1001, 1002, ...] SELECT DISTINCT ON (work_item_type_id) work_item_type_id, enabled FROM work_item_type_visibilities WHERE namespace_id = ANY(ARRAY[1, 5, 12, 45]) AND work_item_type_id = ANY(ARRAY[1001, 1002, ...]) AND ( namespace_id = 45 -- exact match always applies OR ( propagate = true -- or it propagates from ancestor AND array_position(ARRAY[1, 5, 12, 45], namespace_id) < array_position(ARRAY[1, 5, 12, 45], 45) ) ) ORDER BY work_item_type_id, array_position(ARRAY[1, 5, 12, 45], namespace_id) DESC -- closest ancestor wins ``` **Why this is fast:** - Single query for all 40 types - Uses indexes on `namespace_id` and `work_item_type_id` - `traversal_ids` is already in memory (part of namespace record) - Maximum of 40 types × hierarchy depth rows to scan (typically \< 1000 rows) - `DISTINCT ON` ensures we get exactly one result per type **Handling missing overrides:** - If no row exists after resolution → type is enabled (default behavior) - Application layer handles this **array_position performance concerns:** If we want to get rid of the `array_position` function call we could pass the namespace IDs with index. Could be something along the lines of this: ```sql SELECT DISTINCT ON (work_item_type_id) work_item_type_id, enabled FROM work_item_type_visibilities w CROSS JOIN LATERAL ( SELECT rank FROM (VALUES (45, 0), (12, 1), (5, 2), (1, 3) # see reversed order ASC rank ) AS h(ns_id, rank) WHERE h.ns_id = w.namespace_id ) h WHERE w.work_item_type_id = ANY(ARRAY[1001, 1002, ...]) AND ( w.namespace_id = 45 OR (w.propagate = true AND h.rank > 0) ) ORDER BY w.work_item_type_id, h.rank; ``` ### Mutation Operations **THIS namespace only:** ```ruby WorkItemTypeVisibility.upsert( namespace_id: namespace.id, work_item_type_id: type.id, enabled: true, # for enabled, false for disabled propagate: false # don't propagate to children ) ``` **ALL existing children:** ```ruby # 1. Set parent override WorkItemTypeVisibility.upsert( namespace_id: namespace.id, work_item_type_id: type.id, enabled: true, # for enabled, false for disabled propagate: true # propagate to children ) # 2. Remove conflicting descendant overrides (async job allowed) WorkItemTypeVisibility .joins(:namespace) .where(work_item_type_id: type.id) .where("namespaces.traversal_ids @> ARRAY[?]::bigint[]", namespace.id) .where.not(namespace_id: namespace.id) .delete_all ``` The deletion is fast because: - Index on `namespaces.traversal_ids` makes the `@>` (contains) operator efficient - We're only deleting rows for a single type - Even with thousands of descendants, should be fast **NEW children only:** ```ruby WorkItemTypeVisibilityDefault.upsert( namespace_id: namespace.id, work_item_type_id: type.id, enabled: action == :enable ) ``` #### Special case: "Disable for THIS, Enable for children"** A critical requirement is supporting the case where a type is disabled for the current namespace but enabled for its children. For example, disabling "Bug" at the root level but enabling it for all projects. This is achieved through **two separate mutation calls**: ```graphql # Call 1: Disable for THIS namespace only mutation { workItemAvailabilityToggle( input: { namespaceId: "gid://gitlab/Namespace/123" workItemTypeId: "gid://gitlab/WorkItems::Type/5" action: DISABLE scope: THIS } ) } # Call 2: Enable for ALL children mutation { workItemAvailabilityToggle( input: { namespaceId: "gid://gitlab/Namespace/123" workItemTypeId: "gid://gitlab/WorkItems::Type/5" action: ENABLE scope: ALL_CHILDREN } ) } ``` **Backend handling:** When processing `scope: ALL_CHILDREN`, the mutation checks if there's already a non-propagating override at the parent namespace (ruby pseudo code): ```ruby when :ALL_CHILDREN existing_override = WorkItemTypeVisibility.find_by( namespace_id: namespace_id, work_item_type_id: work_item_type_id, propagate: false ) if existing_override # Parent has explicit "this only" setting - keep it unchanged # Create overrides at immediate children instead namespace.children.find_each do |child| WorkItemTypeVisibility.upsert( namespace_id: child.id, work_item_type_id: work_item_type_id, enabled: action == :enable, propagate: true ) end else # Normal case: set parent with propagate: true WorkItemTypeVisibility.upsert( namespace_id: namespace_id, work_item_type_id: work_item_type_id, enabled: action == :enable, propagate: true ) end # Clean up any conflicting deeper overrides (async allowed) WorkItemTypeVisibility .joins(:namespace) .where(work_item_type_id: work_item_type_id) .where("namespaces.traversal_ids @> ARRAY[?]::bigint[]", namespace_id) .where.not(namespace_id: namespace_id) .delete_all ``` **How resolution works for this case:** - **Parent namespace queries visibility:** - `traversal_ids = [parent_id]` - Finds override: `enabled: false, propagate: false` - Type is **disabled** ✅ - **Child namespace queries visibility:** - `traversal_ids = [parent_id, child_id]` - Finds two overrides: - Parent: `enabled: false, propagate: false` → doesn't propagate, skip - Child: `enabled: true, propagate: true` → applies - Type is **enabled** ✅ - **Grandchild namespace queries visibility:** - `traversal_ids = [parent_id, child_id, grandchild_id]` - Finds child's override: `enabled: true, propagate: true` - Child propagates to descendants → Type is **enabled** ✅ The `propagate: false` flag on the parent's override ensures it doesn't apply to descendants, while explicit overrides at the child level (with `propagate: true`) become the effective rule for that subtree. **Performance consideration:** For namespaces with many immediate children, creating overrides at each child can run async but doesn't need to. For typical cases (1-10 children), this completes instantly. #### Namespace creation When creating a new namespace, we materialize applicable future rules as regular overrides (ruby pseudo code): ```ruby def apply_visibility_defaults_to_new_namespace(namespace) ancestor_ids = namespace.traversal_ids[0..-2] # exclude self # Find closest ancestor default rule per type (single query) applicable_defaults = WorkItemTypeVisibilityDefault .where(namespace_id: ancestor_ids) .select('DISTINCT ON (work_item_type_id) *') .order(Arel.sql(" work_item_type_id, array_position(ARRAY[#{ancestor_ids.join(',')}]::bigint[], namespace_id) DESC ")) # Bulk insert as regular overrides overrides = applicable_defaults.map do |default| { namespace_id: namespace.id, work_item_type_id: default.work_item_type_id, enabled: default.enabled, propagate: false # these apply only to the new namespace } end WorkItemTypeVisibility.insert_all(overrides) if overrides.any? end ``` This materializes future rules as regular overrides, so resolution doesn't need to know about the `work_item_type_visibility_defaults` table at all. ### Why This Works 1. **Storage is sparse** - We only store a low percentage of the theoretical namespace × type combinations (only actual overrides) 2. **Reads are fast** - Single query using indexed lookups on `traversal_ids` and `work_item_type_id` 3. **Writes are acceptable** - Bulk operations can run async; individual updates are instant 4. **Namespace moves are free** - Visibility data is keyed by `namespace_id` only; `traversal_ids` is never stored in visibility tables 5. **Clean semantics** - Closest ancestor wins, which matches user mental model of inheritance ## 7. Alternative Approaches Considered ### Event-Sourced Visibility Tree (ESVT) Store all visibility changes as an append-only event log with timestamps. **Why rejected:** - Table grows indefinitely (requires compaction strategy) - Resolution requires temporal queries (find most recent event per type per ancestry path) - More complex SQL (`DISTINCT ON` with timestamp ordering) - Audit trail is valuable, but not worth the read performance cost - We can achieve audit history with a separate audit table if needed ### Policy Snapshot Inheritance (PSI) Pre-materialize effective visibility state for all namespace × type combinations at write time. **Why rejected:** - Potential storage of thousands of namespaces × 40 types = hundreds of thousands of rows - Most rows would be redundant (storing inherited defaults) - Write amplification is severe - changing a root namespace setting requires updating thousands of rows - Index bloat affects all queries, not just visibility queries - We're optimizing for a problem we don't have (read performance is already good enough with sparse storage) The sparse inheritance model strikes the right balance for our use case: deep hierarchies, bulk resolution, and read-heavy workload. ## 8. Rollout plan The frontend team can already start implementing the actions in the three dot menu and sending the correct mutations. The backend team can add a mock implementation of the mutation. These options should be hidden behind a separate feature flag so we have the option to hide the visibility independently from the work on configurable work item types in general. Consuming the `enabled` field is transparent here. The backend can then add the additional visibility resolution layer in the provider and make the mutations work properly. ## 9. Summary I'm proposing a **sparse inheritance model with ancestor resolution** for work item type visibility controls: - **Three independent visibility actions** (THIS, ALL_CHILDREN, NEW_CHILDREN) map cleanly to database operations - **Storage is minimal** - only explicit overrides are stored, not inherited defaults - **Reads are fast** - single query leveraging `traversal_ids` and indexed lookups - **Writes are acceptable** - bulk operations can be async; individual updates are instant - **Namespace moves are free** - no visibility data needs rewriting - **Clean semantics** - closest ancestor wins, matching user expectations The `work_item_settings` table provides a global toggle and is designed for future extensibility. The dual `organization_id`/`namespace_id` columns support both SaaS and self-managed deployments during our migration period.