Related to https://gitlab.com/gitlab-org/gitlab-caproni/-/merge_requests/94 ----- `GitLabGrafanaDashboard` cannot render any grafonnet dashboard on 1.20.0. `g.dashboard.new`, the first call every grafonnet dashboard makes, fails to evaluate, so the CRD added in 1.16.0 has no working path at all. The sample in [`docs/dashboards-grafonnet.md`](https://gitlab.com/gitlab-com/gl-infra/sandbox/metropolis/-/blob/main/docs/dashboards-grafonnet.md) reproduces it verbatim against `operator:v1.20.0`. <:robot:> ## Reproduce Apply the documented sample unchanged: ```yaml apiVersion: metrics.gitlab.com/v1alpha1 kind: GitLabGrafanaDashboard metadata: name: registry-overview spec: jsonnet: | local g = import 'github.com/grafana/grafonnet/gen/grafonnet-latest/main.libsonnet'; g.dashboard.new('Registry Overview') + g.dashboard.withUid('registry-overview') ``` ``` $ kubectl describe gitlabgrafanadashboard registry-overview Reason: RenderFailed Message: rendering dashboard: evaluating dashboard jsonnet: RUNTIME ERROR: import not found: "./dashboard/annotation.libsonnet" (imported from "jsonnet/deps/vendor/github.com/grafana/grafonnet/gen/grafonnet-v11.4.0/custom/dashboard.libsonnet") jsonnet/deps/vendor/github.com/grafana/grafonnet/gen/grafonnet-v11.4.0/custom/dashboard.libsonnet:67:4-45 $ jsonnet/deps/vendor/github.com/grafana/grafonnet/gen/grafonnet-v11.4.0/dashboard.libsonnet:596:4-39 $ jsonnet/deps/vendor/github.com/grafana/grafonnet/gen/grafonnet-v11.4.0/main.libsonnet:14:14-42 object <anonymous> dashboard.jsonnet:1:2-88 ``` ## The file is embedded. It is the relative import that fails The obvious reading is that `jb install` did not fetch the `custom/` subtree, so I checked by importing the "missing" file directly on its JPath instead of letting `custom/dashboard.libsonnet` reach it relatively. Three probe resources, same operator, same image: 1. `import '.../custom/dashboard.libsonnet'` fails with `import not found: "./dashboard/annotation.libsonnet"`, as above. 1. `import '.../custom/dashboard/annotation.libsonnet'` **resolves and evaluates**. It then fails with `Attempt to use super when there is no super class` at `annotation.libsonnet:29:5-10`, which is what that file does when imported standalone. The import itself worked. 1. `import '.../custom/query/prometheus.libsonnet'` also resolves, failing only with `couldn't manifest function as JSON`. So `jsonnet/deps/vendor/.../custom/dashboard/annotation.libsonnet` is present in `CatalogFS` and reachable through the `jsonnet/deps/vendor` JPath. What fails is resolving the same file relatively, from `custom/dashboard.libsonnet`, which imports it as `./dashboard/annotation.libsonnet`. That points at `embedImporter` in [`metricscatalog/jsonnetvm.go`](https://gitlab.com/gitlab-com/gl-infra/sandbox/metropolis/-/blob/main/metricscatalog/jsonnetvm.go) rather than at vendoring. `//go:embed all:jsonnet/deps` is recursive, and `Import` does try `path.Dir(importedFrom)` before the JPaths, so the intent is right, but something in that first attempt is not landing on the path the JPath lookup finds. The per-candidate `cache`, which memoises non-existence, is worth a look given the importer outlives a single evaluation. Reading `custom/dashboard.libsonnet` upstream, the three imports at the bottom are the whole failure surface, and none of them resolve today: ```jsonnet + (import './dashboard/annotation.libsonnet') + (import './dashboard/link.libsonnet') + (import './dashboard/variable.libsonnet') ``` ## The retry behaviour is the more damaging half A `GitLabGrafanaDashboard` that fails to render is requeued without an effective backoff cap, and the resulting churn starves the `GitLabServiceMetrics` controller. Observed on a rig with five `GitLabServiceMetrics` and one failing `GitLabGrafanaDashboard`, mid-way through a dashboard-delivery switch to `globalConfig.dashboards.grafanaOperator`: three services completed the switch, then `gitaly` and `webservice` stopped reconciling entirely and stayed that way for minutes. Deleting the failing `GitLabGrafanaDashboard` unblocked them, and `webservice` converged within seconds of the delete. This is arguably separable from the render bug: any persistently failing render, including a user's own syntax error, can currently stall unrelated `GitLabServiceMetrics` reconciles. Happy to split it out if you would rather track it on its own. ## Suggested checks 1. `find metricscatalog/jsonnet/deps/vendor/github.com/grafana/grafonnet -name '*.libsonnet'` after `mise run jsonnet-deps`, to confirm from the build side what the probes show from the running operator. 1. A unit test in `metricscatalog` that evaluates `g.dashboard.new('x')` through `CatalogFS`. The golden tests added in 1.13.0 cover the `GitLabServiceMetrics` path, which renders through the runbooks dashboard libraries and never touches grafonnet's `custom` tree, so nothing currently exercises this. 1. `.goreleaser.yml`'s `before.hooks` runs only `go mod tidy`, while `mise run build-operator` declares `depends = ["jsonnet-deps"]`. The probes say the deps are present in the published image, so this is not the cause here, but the release path getting them looks incidental rather than declared. </:robot:>