# Spec: first-class Backstage models for the Engineering Catalog ## Problem The Engineering Catalog's Backstage entities are a lossless re-encoding of `runbooks/services/service-catalog.yml`, shaped by the current sync direction (runbooks is SSOT, fidelity verified by round-trip diffing). This produced second-class representations: - Arrays stuffed into comma-separated annotation strings (`gitlab.com/teams`, `gitlab.com/components`, `gitlab.com/sub-components`) - Integers as string annotations (`gitlab.com/sla-weighting`) - Structured records flattened into `metadata.links[]` with title-prefix conventions (`Logs: X`, `Alert: Y`, `Project: Z`) that consumers must parse back - The tier encoded as a tag (`tier-sv`) - No schema for any of it: generated entities are validated only against the generic upstream catalog-info schema, and (bug) the CI validation glob `data/*/*.yaml` misses `data/entities/services/<name>/service.yaml` entirely, so service entities are not validated at all today To make the catalog the SSOT for service information (with `service-catalog.yml` later generated FROM it), the model must become first-class: typed, structured, schema-enforced, and losslessly complete. ## Goals 1. Replace annotation/link-encoded org data with a structured, versioned `spec.gitlab` block on Component, Resource, and Group entities. 2. Introduce a well-defined JSON Schema for the extended model, enforced in catalog CI (and fix the validation coverage gap). 3. Adapt the runbooks -> catalog sync (Jsonnet pipeline) to emit the new model, with round-trip verification updated accordingly. 4. Keep the portal exposing the same information in the UI. Human-facing links are projected from the model at ingestion time by a portal-side processor, not committed to the catalog (decided). ## Non-goals (follow-up tickets) - Turning the sync around (generating `service-catalog.yml` / `teams.yml` from the catalog). This ticket makes it possible; it does not do it. - Introducing System/Domain entities for product stages/sections (separate modeling decision; Groups must remain for ownership regardless). - Adopting third-party plugins (Grafana, PagerDuty) and emitting their annotation contracts. - Portal write capabilities (catalog stays readonly). - Migrating the API consumer (sqlmesh-catalog): separate follow-up, see "API consumer impact" below. ## Design ### 1. Target model Built-in kinds and `apiVersion: backstage.io/v1alpha1` are kept. All org-specific data moves into one org-namespaced block: `spec.gitlab`. Rationale: upstream kind schemas are open (unknown `spec` fields validate, are stored, and are served by the API), annotations are string-only by design, and custom kinds are flagged high-risk upstream and would forfeit the standard Component/Resource UI (see References). Component/Resource entities: ```yaml apiVersion: backstage.io/v1alpha1 kind: Component # or Resource, unchanged classification via mappings.libsonnet metadata: name: api description: GitLab API tags: [tier-sv, rails, kubernetes] # unchanged, incl. derived tier-* tag annotations: backstage.io/edit-url: <url> # kept: plugin-facing keys only backstage.io/source-location: url:<url> # kept # NOTE: no metadata.links in committed YAML; links are derived at ingestion # by a portal-side processor (see 1a). Hand-authored links remain allowed # and are preserved by the processor's merge semantics. spec: type: service lifecycle: production owner: group:default/g_source_code dependsOn: [component:default/postgres, resource:default/redis] # unchanged gitlab: tier: sv # enum: sv|lb|stor|db|inf serviceLabel: API # was gitlab.com/label annotation slaWeighting: 5 # integer, was string annotation teams: # was CSV annotation; real Group refs - group:default/g_create - group:default/g_deploy components: [rails, workhorse] # was CSV annotation; SLI names, not entity refs subComponents: [fluentd, pubsubbeat] # was CSV annotation projects: - https://gitlab.com/gitlab-org/gitlab documents: architecture: https://... design: https://... readinessReview: https://... sreGuide: https://... developerGuide: https://... security: https://... service: [https://...] # array, matching source observability: grafanaFolder: abc123 primaryGrafanaDashboard: api-main/api-overview # slug, not constructed URL sentrySlug: gitlab/gitlabcom/... gitlabDashboard: https://... logging: - { name: Rails, permalink: https://log.gprd... } alerts: - { name: ..., link: https://... } ``` Group entities gain a `spec.gitlab` block carrying the teams.yml attributes, schema-complete for every key in the current teams.yml schema (including rarely-used ones: `manager`, `manager_slack`, `slack_group`, `alerts`, `assign_capacity_planning_issues`, `engagement_policy`, `label`), so teams.yml can later be regenerated losslessly: ```yaml spec: type: stage-group parent: group:default/s_create gitlab: productStageGroup: source_code slackChannel: g_create slackAlertsChannel: feed_alerts_create sendSloAlerts: true sendErrorBudgetWeekly: false slackErrorBudgetChannel: cd-section costOwner: SUP-ORG-262 # Workday code or NOT_ASSIGNED; was gitlab.com/cost-owner annotation oncallSchedule: https://gitlab.pagerduty.com/... pagerdutyService: GitLab Production ignoredComponents: [graphql_query] ``` Notes: - Slug-like source values (`primaryGrafanaDashboard`, `sentrySlug`) are stored as source slugs; URL construction moves into the links projection. - `spec.dependsOn` is already first-class and stays. - All data is GREEN today and stays GREEN; the schema review gate must reject any future non-GREEN field. ### 1a. Links: derived at ingestion, not committed `metadata.links` is the one encoding with standard UI value, so the links surface stays, but the committed YAML does NOT contain it. A catalog processor in the portal backend (`CatalogProcessor.postProcessEntity`, standard extension point, works with the readonly catalog) derives `metadata.links` from `spec.gitlab` at ingestion time and merges them into the final entity, preserving any hand-authored links. Consequences: - The committed model is canonical-only; drift between `spec.gitlab` and links is structurally impossible, including after the sync turnaround when entities are edited by hand/MR. - Derived links land in the final entity, so they flow to the standard UI, search, and the REST API automatically. API consumers get full entities: canonical `spec.gitlab` plus derived links. - Projection logic lives in TypeScript in the portal repo; schema changes to `spec.gitlab` require a coordinated processor update. - The same processor module is the natural later home for emitting `spec.gitlab.teams[]` relations (upstream pattern, backstage#33695). Link derivation, with the schema-sanctioned `type` field replacing title prefixes: | type | source | |---|---| | `project` | spec.gitlab.projects[] | | `docs` | spec.gitlab.documents.* | | `logs` | spec.gitlab.observability.logging[] | | `alert` | spec.gitlab.observability.alerts[] | | `dashboard` | primaryGrafanaDashboard (URL constructed), gitlabDashboard | | `error-tracking` | sentrySlug (URL constructed) | Consumers must never parse link titles again; `spec.gitlab` is the truth, links are display, computed in exactly one place. ### 2. Schema - Per-kind schema files in the catalog repo (`schemas/component.schema.json`, `resource`, `group`, plus shared definitions): each an `allOf` of (a) the upstream kind schema and (b) our `spec.gitlab` definitions for that kind (types, required fields, enums for `tier` and `lifecycle`, URL formats, `costOwner` pattern `^(SUP-ORG-\d+|NOT_ASSIGNED)$`, entity-ref pattern for `teams[]`). Per-kind files give precise editor completion via the modeline below. - **Self-contained files via yaml-language-server modeline.** Every generated YAML file carries a schema pointer in its header comment block (kind-appropriate per output directory): ```yaml # AUTOGENERATED: DO NOT EDIT THIS FILE BY HAND - YOUR CHANGES WILL BE OVERWRITTEN # yaml-language-server: $schema=https://gitlab-com.gitlab.io/gl-infra/engineering-catalog/schemas/v1/component.schema.json ``` Editors validate and autocomplete each file standalone; this matters most after the turnaround, when humans edit entities by hand. It is a comment, so it never enters entity data. Do NOT add `$schema` as a real YAML root key (it would ride into stored entities and API responses, and risks envelope validation). - **Publishing**: `schemas/` published via GitLab Pages under versioned paths (`/schemas/v1/...`), same pattern runbooks uses for its schema docs. - **Pinning**: vendor the upstream kind schemas from `@backstage/catalog-model` at the portal's Backstage version instead of referencing json.schemastore.org (which drifts). Refresh during Backstage upgrades. - **Versioning**: additive-only evolution (mirroring the runbooks schema policy), version marker in the schema `$id`; deprecations get `DEPRECATED:` description prefixes. - **CI (catalog repo)**: replace the `check-jsonschema ... data/*/*.yaml` job with a glob covering ALL generated entity files, including `data/entities/services/*/service.yaml` (fixes the existing coverage gap), validating against the repo schemas (explicit `--schemafile` per kind, no network dependency). Add a drift check asserting each file's modeline points at the schema version CI validated with. - **Portal-side (optional, defense in depth)**: a processor using `entityKindSchemaValidator` to reject malformed `spec.gitlab` at ingestion. Given the portal only ingests our CI-validated YAML, this is hardening, not a launch requirement. ### 3. Sync pipeline changes (catalog repo, Jsonnet) - `lib/annotations.libsonnet`: shrinks to plugin-facing annotations only (`backstage.io/edit-url`, `backstage.io/source-location`). The `gitlab.com/*` extraction is replaced by a new spec builder lib for `spec.gitlab`. - `lib/links.libsonnet`: removed for services (Group links from handbook data stay pipeline-emitted for now); the portal processor owns the projection. - `lib/tags.libsonnet`: unchanged (tier tag stays derived; metrics-catalog tags stay). - `generators/entities.libsonnet`: assembles `spec.gitlab`; owner resolution and `mappings.libsonnet` overrides unchanged. - `generators/groups.libsonnet`: adds `spec.gitlab` to Group entities from teams.yml. - Generator output: emit the yaml-language-server modeline in each generated file's header, kind-appropriate per output directory. - Unit tests (`*.test.libsonnet`) updated alongside each lib/generator. - Verification: `scripts/reverse-catalog.py` reads `spec.gitlab` instead of parsing annotations/link titles (net simplification). Round-trip diff against normalized `service-catalog.yml` must stay clean modulo the documented expected differences; this remains the acceptance gate for losslessness. - Regenerate `data/` in the same MR as the pipeline change so the cutover is atomic. ### 4. Portal changes Confirmed current state: `AboutCardWithLinks` renders `metadata.links` flat and reads only `backstage.io/edit-url`; `DependenciesContent` uses relations from `spec.dependsOn`; no portal code reads `gitlab.com/*` annotations. Work items: - **Mandatory: links-projection backend module** (e.g. `plugins/catalog-backend-module-gitlab-model`): `CatalogProcessor.postProcessEntity` deriving typed `metadata.links` from `spec.gitlab` per the table in 1a, merge semantics, no-op for entities without `spec.gitlab` (deploys safely ahead of the data cutover). Unit-tested against fixture entities. - **UI**: new "Service details" entity card (NFS `EntityCardBlueprint`) rendering `spec.gitlab` (tier, SLA weighting, service label, contributing teams as entity links, components/subComponents), filtered on `spec.gitlab` presence; group links in `AboutCardWithLinks` by `link.type` with a flat-list fallback. - **API: no backend change required.** The catalog REST API serves full final entities (committed `spec.gitlab` plus processor-derived links) and filters accept `spec.gitlab.*` paths (e.g. `filter=spec.gitlab.tier=sv`). Unknown spec fields are never stripped (kind schemas are open; validation gates, it does not filter). Caveat: nested object arrays (logging/alerts) are retrievable but not usefully filterable; do not design filters against them. - **Optional (decide)**: schema-validation processor; custom relations for `spec.gitlab.teams[]` (upstream precedent backstage#33695). Both would share the new backend module. Recommendation: follow-ups. ## API consumer impact (sqlmesh-catalog) The API mechanics do not change; the entity payload shape does. The only known API consumer is [`gitlab-com/gl-infra/data/sqlmesh-catalog`](https://gitlab.com/gitlab-com/gl-infra/data/sqlmesh-catalog) (cloud cost attribution). Verified against its source on `main`: - `src/engineering_catalog/backstage_helpers.py` pages `/api/catalog/entities/by-query?filter=kind=<Kind>` with a static `catalog:read` bearer token. Endpoint, filter syntax, pagination: unaffected. - `models/engineering_catalog/dim_backstage_services.py` reads `kind`, `metadata.{name,namespace,description,tags}`, `spec.{type,owner}`: all core-model fields, unaffected. - `models/engineering_catalog/dim_backstage_teams.py` additionally reads `metadata.annotations["gitlab.com/cost-owner"]` from Group entities into `cost_owner_workday_group`. **This is the single breaking read site**: the value moves to `spec.gitlab.costOwner`. Required consumer change (dual-read form, deployable before the cutover): ```python cost_owner = (spec.get("gitlab") or {}).get("costOwner") \ or annotations.get(COST_OWNER_ANNOTATION) ``` Sequencing nuance: both dims are SCD Type 2 with `cost_owner_workday_group` in the change-tracked columns. If the cutover lands before the consumer fix, the next daily run reads NULL for every group and writes a spurious row version for every Group (and, denormalized, every service), then a second one once fixed. **The dual-read must be deployed before the cutover MR merges**; the legacy fallback can be removed any time after. Handled as a follow-up ticket in sqlmesh-catalog (decision: adjust consumers, no compatibility shims in Backstage). For completeness, changes any other consumer would see: all `gitlab.com/*` annotations disappear (moved per the tables above; `gitlab.com/teams` values additionally change representation from runbooks team names to Group entity refs); link titles become display-only and unstable (use `link.type` or `spec.gitlab`); filter/field expressions referencing `metadata.annotations.gitlab.com/*` silently match nothing and must move to `spec.gitlab.*` paths; raw-YAML readers additionally lose `metadata.links` entirely (projection exists only in served entities). Entity etags rotate once at cutover; `metadata.uid` values persist. ## Rollout / MR sequencing ``` runbooks (SSOT, unchanged) catalog repo portal backend service-catalog.yml --\ GitLab provider ingests teams.yml ---+--> import-data.sh --> Jsonnet pipeline --> data/ --> links processor derives handbook yml --/ emits spec.gitlab ^ | metadata.links from metrics-catalog -/ (no links emitted | | spec.gitlab (merge) for services) | v | CI: layered JSON Schema v validates ALL entities final entity: spec.gitlab round-trip diff green + typed links -> UI + API ``` 1. **MR 1 (catalog): schema + CI.** Per-kind schemas under `schemas/` (target shape), vendored upstream schemas, fixed validation glob, modeline drift check, Pages publishing of `/schemas/v1/`. 2. **MR 2 (portal): links processor + UI.** Backend module (merge semantics, no-op without `spec.gitlab`), Service details card, links grouped by `type`. Inert against current data. Must be DEPLOYED TO PRODUCTION (manual deploy) before MR 3 merges, otherwise service links vanish from the UI at cutover. 3. **MR 3 (catalog): pipeline cutover.** New spec builder, links emission removed for services, updated tests and `reverse-catalog.py`, regenerated `data/`, round-trip diff green, docs updated (`docs/backstage-entity-model.md`, `docs/service-catalog-migration.md`, `CLAUDE.md`). 4. **Follow-ups filed:** sqlmesh-catalog dual-read (MUST deploy before MR 3, see above); sync turnaround generator; System/Domain decision; optional schema-validation + teams-relations processors; incident.io importer and label-reconciliation touchpoints at turnaround time. Hard ordering constraints before MR 3 merges: (a) MR 2 deployed to production, (b) sqlmesh-catalog dual-read deployed. No dual-write/transition period is needed otherwise: the only consumers of the current annotations and link titles are `reverse-catalog.py` (updated in MR 3), sqlmesh-catalog's single cost-owner read, and nothing in the portal. ## Test plan - Jsonnet unit tests per lib/generator (existing pattern). - Schema validation in CI over all generated files (assert validated-file count > 140 to prove glob coverage). - Round-trip: `reverse-catalog.py` output diffs clean against normalized `service-catalog.yml` (plus a new teams.yml reverse check for Group data). Reads committed YAML, so independent of the portal processor. - Portal processor unit tests: fixture entity with full `spec.gitlab` produces expected typed links; entity without `spec.gitlab` passes through untouched; hand-authored links survive the merge. - Portal: local run against regenerated data; probe of an entity page asserting derived links render and the Service details card shows tier/teams; existing e2e suite. - API spot checks: `GET /api/catalog/entities/by-name/component/default/api` returns `spec.gitlab` intact AND derived `metadata.links`; `filter=spec.gitlab.tier=sv` returns the expected set. ## Acceptance criteria - [ ] No `gitlab.com/*` annotations remain in generated entities; all org data lives under `spec.gitlab` typed per schema - [ ] All generated entity files validate in CI against the layered per-kind schemas (services included) - [ ] Every generated file carries a yaml-language-server modeline pointing at its published schema; CI asserts modeline/schema-version consistency; schemas published via Pages - [ ] Committed service entities carry no `metadata.links`; served entities carry processor-derived links with `type`; no consumer parses title prefixes - [ ] Round-trip fidelity preserved for service-catalog.yml AND teams.yml data - [ ] Portal renders at least the same information as before (links, about, dependencies) plus the Service details card - [ ] Catalog docs and CLAUDE.md describe the new model, including the ingestion-time links projection - [ ] No catalog REST API changes; full entities (incl. `spec.gitlab`) confirmed served and filterable - [ ] sqlmesh-catalog follow-up created and dual-read deployed before the cutover MR merges ## Decisions to confirm 1. **Extension block name**: `spec.gitlab` (recommended: short, Jsonnet/TS/filter-path friendly; vendor-named so upstream collision risk is negligible) vs `spec['gitlab.com/catalog']` (collision-proof, hostile ergonomics). 2. **Schema strategy in MR 1**: go straight to target-shape schema (recommended) vs interim schema for current model. 3. **teamsYamlExclusions**: model excluded teams' data now or defer to the turnaround ticket (recommended: defer, record there). 4. **Optional processors** (schema validation, teams relations): follow-ups (recommended) or in scope. The links-projection processor itself is decided and in scope. ## References - https://backstage.io/docs/features/software-catalog/extending-the-model (extension ladder; custom kinds "high risk") - https://backstage.io/docs/features/software-catalog/descriptor-format (annotations string-only; links `type`) - https://github.com/backstage/backstage/blob/master/packages/catalog-model/src/schema/kinds/Component.v1alpha1.schema.json (open spec schema) - https://backstage.io/docs/features/software-catalog/software-catalog-api (full final entities served; arbitrary-path filters) - https://backstage.io/docs/reference/catalog-model.entitykindschemavalidator/ - https://backstage.io/docs/frontend-system/building-plugins/common-extension-blueprints (NFS entity cards) - https://github.com/backstage/backstage/issues/33695 (multi-owner pattern: extended spec array + emitted relations) - https://roadie.io/blog/kinds-and-types-in-backstage/ and https://roadie.io/docs/catalog/validator/ (precedent) :robot: AI-generated content.