revert(gkg): disable system_notes in orbit-prd (datalake CH OOM)

Michael Angelo Riverarequested to merge
arivera/gkg-prd-disable-system-notes into main

Disables system_notes in orbit-prd. Partial revert of !2194 (merged) (keeps image 0.71.0 and chart 1.1.0; stg stays enabled for fixing).

Why: during the v56 backfill the entity.systemnote handler runs a JOIN/aggregation over siphon_system_note_metadata that repeatedly blows the datalake ClickHouse 108 GiB query-memory limit (Code 241, MEMORY_LIMIT_EXCEEDED), even after page-streaming shrinks the block size. Retries exhaust and system-note indexing messages are terminated, so v56 would gate with incomplete system_notes data on the largest namespaces. GKG pods themselves are healthy (0 restarts/OOM) — the ceiling is on the source datalake, which the heavy query was also pushing into collateral failures for other tables.

Sample error 
ERROR handler failed  handler="entity.systemnote"
  processing failed: datalake stream failed: ... Code: 241.
  (total) memory limit exceeded: would use 112.38 GiB ... maximum: 108.00 GiB
  ... reading siphon_system_note_metadata ... FillingRightJoinSide ... (MEMORY_LIMIT_EXCEEDED)

Follow-up: optimize the systemnote datalake query (reduce the JOIN/aggregation memory footprint) before re-enabling in prd.

Files changed
  • services/gkg/env/orbit-prd/values.yaml — remove features.system_notes: true

Merge request reports