Draft: Validate repository-backed skills in the AI Catalog (proof of concept, do not merge)

Sam Beckhamrequested to merge
feature/ai-catalog-skill-validation-poc into feature/ai-catalog-repo-backed-skills-poc

What does this MR do and why?

Adds validation for repository-backed AI Catalog skills, and makes publishing safe to fail.

Builds on !256407 (closed) and targets that branch, so the diff here is only the validation work.

The bug this starts from. Publishing reconciled catalog items against the SKILL.md files at default-branch HEAD with plain desired-state sync keyed by path, and the scanner dropped anything that failed validation — logged, returned nil, discarded by filter_map. So a file that failed validation was indistinguishable from a file that had been deleted. A typo in YAML front matter soft-deleted the catalog item, its versions, and every item consumer that had enabled the skill. A skill enabled across five projects was revoked in all five, with an AppJsonLogger warning as the only trace.

One validator instead of three. Validation existed in three places and no two agreed, nor did any match the Agent Skills specification they all claim to follow. The form required lowercase alphanumerics with single hyphens and 3–64 characters; the publisher allowed 3–255 with no character check at all and treated name as optional, falling back to the directory name. So My Skill! was refused by the form and published without complaint. Gitlab::Ai::Catalog::Skills::Validator now holds the rules, encoded as a vendored JSON schema, and the publisher, the scaffolder and a new merge request check all call it.

Three reconcile outcomes, not two. Validation failure keeps the item and its last valid version and marks it stale, recording the reason and the rejected commit. A path genuinely absent is still deleted. A truncated scan suppresses deletions entirely — the protection a failed scan already had, which a scan that simply hit MAX_SKILLS did not.

Two audiences, opposite needs. The author broke the file but is in the merge request, not the catalog, so a widget meets them there; it renders nothing at all on merge requests that touch no SKILL.md. The consumer is being served something older than the repository holds and usually cannot fix it, so they get a banner, an "Out of date" badge, and the published commit shown beside the rejected one.

Nothing blocks. No push check and no merge gate, matching CI config: nothing in lib/gitlab/checks/ parses .gitlab-ci.yml and there is no mergeability rule for config validity either. Once publish-time failure is non-destructive, the earlier checks are free to be advisory.

Two decisions are recorded as ADRs in docs/adr/ because they are hard to reverse and will look arbitrary later: stale-rather-than-delete, and vendoring the specification rather than fetching it.

Known gaps

  • A skill that has never published has no item to attach a failure to, because Ai::Catalog::Item validates latest_version. The merge request check catches most of these before they land; what to do with the rest is a UX question. Rejections are keyed by path throughout so that decision stays open.
  • Renaming a skill directory still deletes and recreates the item. Inherited from the parent MR, not addressed here.
  • A pre-existing failure on the parent branch: ee/spec/models/ai/catalog/item_spec.rb:256 asserts item_type has four values, but skill: 5 was added without updating it. Left alone — it belongs on the parent branch.

References

Screenshots or screen recordings

Not yet captured. This is the part that matters most for the UX conversation, so they need adding before review.

Before After

Three surfaces to capture:

  1. Skill detail page with a stale skill — banner plus the "Not published" row beside "Last commit".
  2. Catalog list with an "Out of date" badge.
  3. Merge request widget, both states: all skills conformant, and skills with problems.

How to set up and validate locally

  1. Enable the AI Catalog and create a project with a skill:

    rake "gitlab:ai_catalog:publish_skills[group/project]"

  2. Commit a conformant skill to the default branch at skills/cosign/SKILL.md:

    ---
name: cosign
description: Signs blobs with Sigstore cosign.
metadata:
  gitlab.visibility: private
---

## Instructions

  3. Confirm it publishes, then enable it on a project so it has an item consumer.

  4. Break it — change name to Cosign — and merge that to the default branch.

  5. Confirm the catalog item still exists, still has its consumer, and is now marked out of date with the reason and the rejected commit shown.

  6. Open a merge request that changes a SKILL.md and confirm the widget reports the problems and says merging is not blocked.

  7. Open any merge request that touches no skills and confirm the widget does not render.

MR acceptance checklist

  • Tests. 1207 backend examples and 1820 frontend in the affected areas. New coverage for the validator, the three-way reconcile, BlobCheck, the merge request resolver, the widget, and a query-count assertion on the new list field.
  • Database. One migration adding sync_status and sync_metadata to ai_catalog_items. Needs database review.
  • Changelog. Deliberately omitted — non-mergeable proof of concept, matching the parent MR.
  • Feature flag. Deliberately omitted — non-mergeable proof of concept for usability testing.
  • Translation. All new user-facing strings are externalised; locale/gitlab.pot regenerated.
  • Documentation. GraphQL reference regenerated. No user documentation, since nothing here ships.
  • Security. New GraphQL resolver checks read_code on the source project on top of the type's read_merge_request.

Merge request reports