Add internal visibility support for AI Catalog finders and model

Jaydip Pansuriyarequested to merge
603251-internal-visibility-finders-model into master

Summary

Adds support for internal visibility on AI Catalog items, gated behind the ai_catalog_internal_visibility feature flag (default off).

AI Catalog items currently support two visibility states via a public boolean column: public (enable by all projects in the organization) and private (enable only within the owning project). As part of the #603251 (closed) this MR introduces a third state, internal - meaning the item can be enabled in any project within the same top-level group hierarchy.

Since a boolean column cannot represent three states, the previous MR (!241436 (merged)) added a visibility enum column (private: 0, internal: 1, public: 2). This MR puts that column to use.

Changes in this MR

  • Dual-write sync callbacks: sync_public_from_visibility keeps the public boolean in sync when visibility is set directly, ensuring backward compatibility until the public column is removed
  • Visibility reduction validation: Prevents reducing visibility (e.g. public → internal, internal → private) when the item has external consumers
  • ProjectItemsFinder: When the feature flag is enabled, returns internal items scoped to the same top-level group alongside public items. When disabled, falls back to the public boolean (internal items have public: false, so they are safely excluded)
  • ItemConsumer validation: Allows enabling internal items in projects/groups within the same top-level group hierarchy
  • Composite index on (organization_id, visibility) WHERE deleted_at IS NULL to support the new query path
  • Feature flag ai_catalog_internal_visibility (user-scoped, wip) to control rollout

Below changes will be part of the next MRs

  • Creation/Updation of internal items
  • Policy changes
  • GraphQL changes

How it works

Visibility FF ON FF OFF
Public Visible to all projects in org Visible to all projects in org
Internal Visible within the top-level group Hidden (treated as private via public: false)
Private Visible only in owning project Visible only in owning project

References

#603251 (closed)

How to set up and validate locally

Click to expand

Validation Steps (GDK)

1. Run the migration

bin/rails db:migrate

2. Set up test data via Rails console

# Find or create a top-level group and projects
group = Group.find 1000000 # use your existing TLG
project_a = group.projects.first
project_b = group.projects.second # another project in same TLG

# A project outside the TLG
outside_project = Project.where.not(namespace_id: group.self_and_descendants.select(:id)).first

user = User.find_by(username: 'your-username')

internal_item = Ai::Catalog::Item.where(project: group.all_projects).last
internal_item.visibility = :internal
internal_item.save!

3. Verify with FF OFF (default behavior)

Feature.disable(:ai_catalog_internal_visibility)

# Internal items should NOT appear in all_available results
finder = Ai::Catalog::ProjectItemsFinder.new(user, project_a, params: { all_available: true })
results = finder.execute
results.include?(internal_item) # => false

4. Verify with FF ON

Feature.enable(:ai_catalog_internal_visibility, user)

# Internal items should appear for project_a
finder_a = Ai::Catalog::ProjectItemsFinder.new(user, project_a, params: { all_available: true })
results_a = finder_a.execute
results_a.include?(internal_item) # => true

# Internal items should also appear for project_b
finder_b = Ai::Catalog::ProjectItemsFinder.new(user, project_b, params: { all_available: true })
results_b = finder_b.execute
results_b.include?(internal_item) # => true

# Internal items should NOT appear for a project outside the TLG
finder_outside = Ai::Catalog::ProjectItemsFinder.new(user, outside_project, params: { all_available: true })
results_outside = finder_outside.execute
results_outside.include?(internal_item) # => false

Database changes

1) ProjectItemsFinder

finder_a = Ai::Catalog::ProjectItemsFinder.new(user, project_a, params: { all_available: true })
results_a = finder_a.execute

When the feature flag is enabled, the finder adds a subquery to resolve internal items within the top-level group hierarchy. This accounts for the additional SubPlan visible in the new query plan compared to the current one.

I have added index and internal items in the postgres.ai session to find the accurate query plan, so above both plan is based on that data.

2) Index index_ai_catalog_items_on_org_id_and_visibility_not_deleted added

CREATE INDEX CONCURRENTLY index_ai_catalog_items_on_org_id_and_visibility_not_deleted
ON ai_catalog_items (organization_id, visibility)
WHERE deleted_at IS NULL;

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Related to #603251 (closed)

Edited by Jaydip Pansuriya

Merge request reports