Skip to content

Document values yaml overrides

Robert Hunt requested to merge document-values-yaml-overrides into main

What does this MR do and why?

This MR adds a process to document all the used values.yaml properties. This is non-exhaustive as there could be unused properties available for the deployment environment.

To make this process less painful, I've written a bash script which uses yq to populate a Markdown file to document:

  • The property path
  • The property value set by the specific values.yaml
  • The location of the values.yaml

There are potentially plenty of future improvements with this script including:

  • Combining the yq usage to improve the performance.
  • Automatically populating the Markdown document for you.
  • Adding line numbers to the source link.

I highly suggest viewing the Markdown document in preview mode rather than reading it as source code 😅.

MR acceptance checklist

  • The correct type labels have been applied to this MR.
  • This MR has been made as small as possible, to improve review efficiency and code quality.
  • This MR has been self-reviewed per the code review guidelines.
  • The changes have undergone manual testing and are functioning as intended.
  • [-] This MR has updated the Chart.yaml version number following SemVer versioning practices.
  • This MR documents any breaking changes in the MR description, and the upgrade path has been documented in the first commit as well as in MR description.

How to set up and validate

Review the Markdown document values and compare against the values.yaml files in the repo.

Example script output: 
./scripts/generate-values-list.sh
Copying docs/templates/values-list.tmpl.md markdown template to docs/values-list.md..

Generating values list...

Found the following values.yaml files to process:
helm-charts/charts/clickhouse/values.yaml
helm-charts/charts/cube/values.yaml
helm-charts/charts/certificates/values.yaml
helm-charts/charts/snowplow/values.yaml
helm-charts/values.yaml

Getting keys from helm-charts/charts/clickhouse/values.yaml...
- Adding deployment.server.image.repository to markdown table
- Adding deployment.server.image.tag to markdown table
- Adding deployment.replicas to markdown table
- Adding deployment.resources.requests.cpu to markdown table
- Adding deployment.resources.requests.memory to markdown table
- Adding deployment.resources.limits.cpu to markdown table
- Adding deployment.resources.limits.memory to markdown table
- Adding pdb.maxUnavailable to markdown table
- Adding pvc.lib.storageRequest to markdown table
- Adding pvc.log.storageRequest to markdown table

Getting keys from helm-charts/charts/cube/values.yaml...
- Adding api.domain to markdown table
- Adding api.env.cubejsDbUser to markdown table
- Adding api.env.cubejsDbSslCa to markdown table
- Adding api.env.cubejsDbSslCert to markdown table
- Adding api.env.cubejsDbSslKey to markdown table
- Adding api.env.cubejsDevMode to markdown table
- Adding api.env.cubejsExternalDefault to markdown table
- Adding api.image.repository to markdown table
- Adding api.image.tag to markdown table
- Adding api.pdb.maxUnavailable to markdown table
- Adding api.replicas to markdown table
- Adding api.resources.requests.cpu to markdown table
- Adding api.resources.requests.memory to markdown table
- Adding api.resources.limits.cpu to markdown table
- Adding api.resources.limits.memory to markdown table
- Adding api.service.type to markdown table
- Adding api.pvc.nodeModules.storageRequest to markdown table
- Adding refreshWorker.pdb.maxUnavailable to markdown table
- Adding refreshWorker.resources.requests.cpu to markdown table
- Adding refreshWorker.resources.requests.memory to markdown table
- Adding refreshWorker.resources.limits.cpu to markdown table
- Adding refreshWorker.resources.limits.memory to markdown table
- Adding refreshWorker.pvc.nodeModules.storageRequest to markdown table
- Adding store.router.env.cubestoreDataDir to markdown table
- Adding store.router.env.cubestoreMetaPort to markdown table
- Adding store.router.env.cubestoreRemoteDir to markdown table
- Adding store.router.image.repository to markdown table
- Adding store.router.image.tag to markdown table
- Adding store.router.pdb.maxUnavailable to markdown table
- Adding store.router.ports to markdown table
- Adding store.router.ports.0.name to markdown table
- Adding store.router.ports.0.port to markdown table
- Adding store.router.ports.0.targetPort to markdown table
- Adding store.router.ports.1.name to markdown table
- Adding store.router.ports.1.port to markdown table
- Adding store.router.ports.1.targetPort to markdown table
- Adding store.router.ports.2.name to markdown table
- Adding store.router.ports.2.port to markdown table
- Adding store.router.ports.2.targetPort to markdown table
- Adding store.router.replicas to markdown table
- Adding store.router.type to markdown table
- Adding store.workers.env.cubestoreDataDir to markdown table
- Adding store.workers.env.cubestoreRemoteDir to markdown table
- Adding store.workers.env.cubestoreWorkerPort to markdown table
- Adding store.workers.image.repository to markdown table
- Adding store.workers.image.tag to markdown table
- Adding store.workers.initRouter.image.repository to markdown table
- Adding store.workers.initRouter.image.tag to markdown table
- Adding store.workers.pdb.maxUnavailable to markdown table
- Adding store.workers.ports to markdown table
- Adding store.workers.ports.0.name to markdown table
- Adding store.workers.ports.0.port to markdown table
- Adding store.workers.ports.0.targetPort to markdown table
- Adding store.workers.replicas to markdown table
- Adding store.workers.type to markdown table
- Adding store.volumeClaims.cubestorePv.requests.storage to markdown table
- Adding deployment.cube.image.repository to markdown table
- Adding deployment.cube.image.tag to markdown table
- Adding deployment.cube.js.externalDefault to markdown table
- Adding deployment.cube.js.scheduledRefreshDefault to markdown table
- Adding deployment.cube.js.devMode to markdown table
- Adding deployment.cube.js.dbPort to markdown table
- Adding deployment.replicas to markdown table
- Adding ingress.google.backendConfig.enabled to markdown table
- Adding ingress.google.backendConfig.connectionDraining.drainingTimeoutSec to markdown table
- Adding ingress.google.backendConfig.healthCheck.port to markdown table
- Adding ingress.google.backendConfig.healthCheck.type to markdown table
- Adding ingress.google.backendConfig.healthCheck.requestPath to markdown table
- Adding ingress.google.backendConfig.healthCheck.timeoutSec to markdown table
- Adding ingress.google.backendConfig.healthCheck.checkIntervalSec to markdown table
- Adding ingress.google.backendConfig.healthCheck.healthyThreshold to markdown table
- Adding ingress.google.backendConfig.healthCheck.unhealthyThreshold to markdown table
- Adding ingress.google.backendConfig.timeoutSec to markdown table
- Adding ingress.google.frontendConfig.enabled to markdown table
- Adding ingress.google.frontendConfig.redirectToHttps.enabled to markdown table
- Adding ingress.google.frontendConfig.sslPolicy to markdown table
- Adding ingress.google.globalStaticIpName to markdown table
- Adding ingress.google.managedCertificate.enabled to markdown table
- Adding ingress.google.securityPolicy.enabled to markdown table
- Adding secrets.apiSecret to markdown table
- Adding secrets.existingSecret to markdown table
- Adding tls.clickhouse.existingSecret to markdown table

Getting keys from helm-charts/charts/certificates/values.yaml...
- Adding ca.commonName to markdown table
- Adding ca.subject.organizations to markdown table
- Adding ca.subject.organizations.0 to markdown table
- Adding ca.subject.organizationalUnits to markdown table
- Adding ca.subject.organizationalUnits.0 to markdown table
- Adding letsencrypt_issuer.enabled to markdown table
- Adding letsencrypt_issuer.acme.email to markdown table
- Adding kafka.replicaCount to markdown table
- Adding kafka.keystore.password to markdown table
- Adding kafka.keystore.existingSecret to markdown table
- Adding snowplow.collector.keystore.password to markdown table
- Adding snowplow.collector.keystore.existingSecret to markdown table

Getting keys from helm-charts/charts/snowplow/values.yaml...
- Adding configurator.domain to markdown table
- Adding configurator.port to markdown table
- Adding configurator.auth.username to markdown table
- Adding configurator.auth.password to markdown table
- Adding configurator.auth.existingSecret to markdown table
- Adding configurator.kafka.topic to markdown table
- Adding configurator.kafka.bad_events_topic to markdown table
- Adding configurator.image.registry to markdown table
- Adding configurator.image.repository to markdown table
- Adding configurator.image.tag to markdown table
- Adding configurator.image.pullPolicy to markdown table
- Adding configurator.pdb.maxUnavailable to markdown table
- Adding configurator.ingress.google.backendConfig.enabled to markdown table
- Adding configurator.ingress.google.backendConfig.connectionDraining.drainingTimeoutSec to markdown table
- Adding configurator.ingress.google.backendConfig.healthCheck.port to markdown table
- Adding configurator.ingress.google.backendConfig.healthCheck.type to markdown table
- Adding configurator.ingress.google.backendConfig.healthCheck.requestPath to markdown table
- Adding configurator.ingress.google.backendConfig.healthCheck.timeoutSec to markdown table
- Adding configurator.ingress.google.backendConfig.healthCheck.checkIntervalSec to markdown table
- Adding configurator.ingress.google.backendConfig.healthCheck.healthyThreshold to markdown table
- Adding configurator.ingress.google.backendConfig.healthCheck.unhealthyThreshold to markdown table
- Adding configurator.ingress.google.backendConfig.timeoutSec to markdown table
- Adding configurator.ingress.google.frontendConfig.enabled to markdown table
- Adding configurator.ingress.google.frontendConfig.redirectToHttps.enabled to markdown table
- Adding configurator.ingress.google.frontendConfig.sslPolicy to markdown table
- Adding configurator.ingress.google.globalStaticIpName to markdown table
- Adding configurator.ingress.google.managedCertificate.enabled to markdown table
- Adding collector.domain to markdown table
- Adding collector.port to markdown table
- Adding collector.keystore.existingSecret to markdown table
- Adding collector.keystore.password to markdown table
- Adding collector.image.registry to markdown table
- Adding collector.image.repository to markdown table
- Adding collector.image.tag to markdown table
- Adding collector.image.pullPolicy to markdown table
- Adding collector.pdb.maxUnavailable to markdown table
- Adding collector.resources.requests.cpu to markdown table
- Adding collector.resources.requests.memory to markdown table
- Adding collector.resources.requests.ephemeral-storage to markdown table
- Adding collector.autoscaling.enabled to markdown table
- Adding collector.autoscaling.minReplicas to markdown table
- Adding collector.autoscaling.maxReplicas to markdown table
- Adding collector.autoscaling.targetCPUUtilizationPercentage to markdown table
- Adding collector.ingress.google.backendConfig.enabled to markdown table
- Adding collector.ingress.google.backendConfig.connectionDrainingEnabled to markdown table
- Adding collector.ingress.google.backendConfig.connectionDraining.drainingTimeoutSec to markdown table
- Adding collector.ingress.google.backendConfig.healthCheckEnabled to markdown table
- Adding collector.ingress.google.backendConfig.healthCheck.port to markdown table
- Adding collector.ingress.google.backendConfig.healthCheck.type to markdown table
- Adding collector.ingress.google.backendConfig.healthCheck.requestPath to markdown table
- Adding collector.ingress.google.backendConfig.healthCheck.timeoutSec to markdown table
- Adding collector.ingress.google.backendConfig.healthCheck.checkIntervalSec to markdown table
- Adding collector.ingress.google.backendConfig.healthCheck.healthyThreshold to markdown table
- Adding collector.ingress.google.backendConfig.healthCheck.unhealthyThreshold to markdown table
- Adding collector.ingress.google.backendConfig.timeoutSec to markdown table
- Adding collector.ingress.google.frontendConfig.enabled to markdown table
- Adding collector.ingress.google.frontendConfig.redirectToHttps.enabled to markdown table
- Adding collector.ingress.google.frontendConfig.sslPolicy to markdown table
- Adding collector.ingress.google.globalStaticIpName to markdown table
- Adding collector.ingress.google.managedCertificate.enabled to markdown table
- Adding collector.ingress.google.securityPolicy.enabled to markdown table
- Adding collector.monitoring.metrics.statsd.enabled to markdown table
- Adding collector.config to markdown table
- Adding enricher.image.registry to markdown table
- Adding enricher.image.repository to markdown table
- Adding enricher.image.tag to markdown table
- Adding enricher.image.pullPolicy to markdown table
- Adding enricher.image.pullSecrets to markdown table
- Adding enricher.pdb.maxUnavailable to markdown table
- Adding enricher.resources.requests.cpu to markdown table
- Adding enricher.resources.requests.memory to markdown table
- Adding enricher.autoscaling.enabled to markdown table
- Adding enricher.autoscaling.minReplicas to markdown table
- Adding enricher.autoscaling.maxReplicas to markdown table
- Adding enricher.autoscaling.targetCPUUtilizationPercentage to markdown table
- Adding enricher.pseudonymization.enabled to markdown table
- Adding enricher.pseudonymization.salt to markdown table
- Adding enricher.config to markdown table
- Adding enricher.iglu to markdown table
- Adding serviceAccount.create to markdown table
- Adding serviceAccount.name to markdown table

Getting keys from helm-charts/values.yaml...
- Adding global.certificates.enabled to markdown table
- Adding global.certificates.letsencrypt_dns_issuer.enabled to markdown table
- Adding global.clickhouse.enabled to markdown table
- Adding global.clickhouse.auth.existingSecret to markdown table
- Adding global.clickhouse.credentials.username to markdown table
- Adding global.clickhouse.credentials.password to markdown table
- Adding global.clickhouse.credentials.existingSecret to markdown table
- Adding global.clickhouse.host to markdown table
- Adding global.clickhouse.ports.secure.http to markdown table
- Adding global.clickhouse.ports.secure.client to markdown table
- Adding global.clickhouse.ports.secure.internal to markdown table
- Adding global.clickhouse.ports.insecure.http to markdown table
- Adding global.clickhouse.ports.insecure.client to markdown table
- Adding global.clickhouse.ports.insecure.internal to markdown table
- Adding global.cube.enabled to markdown table
- Adding global.kafka.enabled to markdown table
- Adding global.snowplow.enabled to markdown table
- Adding global.vector.enabled to markdown table
- Adding global.promtail.enabled to markdown table
- Adding global.kubernetesClusterDomain to markdown table
- Adding global.dns.projectId to markdown table
- Adding global.dns.zoneName to markdown table
- Adding global.tls.enabled to markdown table
- Adding global.tls.certificates.clickhouse.existingSecret to markdown table
- Adding global.tls.certificates.kafka.existingSecret to markdown table
- Adding global.tls.certificates.snowplow_collector.existingSecret to markdown table
- Adding global.prometheus.enabled to markdown table
- Adding global.prometheus-statsd-exporter.enabled to markdown table
- Adding kafka.replicaCount to markdown table
- Adding kafka.pdb.create to markdown table
- Adding kafka.pdb.maxUnavailable to markdown table
- Adding kafka.zookeeper.replicaCount to markdown table
- Adding kafka.logRetentionHours to markdown table
- Adding kafka.auth.clientProtocol to markdown table
- Adding kafka.auth.tls.password to markdown table
- Adding kafka.auth.tls.existingSecrets to markdown table
- Adding kafka.auth.tls.existingSecrets.0 to markdown table
- Adding kafka.auth.tls.existingSecrets.1 to markdown table
- Adding kafka.auth.tls.existingSecrets.2 to markdown table
- Adding kafka.auth.tls.jksTruststore to markdown table
- Adding kafka.auth.tls.jksKeystoreSAN to markdown table
- Adding kafka.auth.tls.endpointIdentificationAlgorithm to markdown table
- Adding kafka.metrics.kafka.enabled to markdown table
- Adding kafka.metrics.kafka.tlsCaCert to markdown table
- Adding certificates.kafka.replicaCount to markdown table
- Adding certificates.kafka.keystore.password to markdown table
- Adding vector.role to markdown table
- Adding vector.podDisruptionBudget.enabled to markdown table
- Adding vector.podDisruptionBudget.minAvailable to markdown table
- Adding vector.podDisruptionBudget.maxUnavailable to markdown table
- Adding vector.customConfig.data_dir to markdown table
- Adding vector.customConfig.api.enabled to markdown table
- Adding vector.customConfig.api.address to markdown table
- Adding vector.customConfig.api.playground to markdown table
- Adding vector.customConfig.sources.snowplow_enriched_events.type to markdown table
- Adding vector.customConfig.sources.snowplow_enriched_events.auto_offset_reset to markdown table
- Adding vector.customConfig.sources.snowplow_enriched_events.group_id to markdown table
- Adding vector.customConfig.sources.snowplow_enriched_events.topics to markdown table
- Adding vector.customConfig.sources.snowplow_enriched_events.topics.0 to markdown table
- Adding vector.customConfig.sources.snowplow_enriched_events.tls.enabled to markdown table
- Adding vector.customConfig.sources.snowplow_enriched_events.tls.verify_certificate to markdown table
- Adding vector.customConfig.sources.snowplow_enriched_events.tls.ca_file to markdown table
- Adding vector.customConfig.sources.snowplow_enriched_events.tls.crt_file to markdown table
- Adding vector.customConfig.sources.snowplow_enriched_events.tls.key_file to markdown table
- Adding vector.customConfig.sources.snowplow_enriched_events.tls.key_pass to markdown table
- Adding vector.customConfig.sources.snowplow_bad_events.type to markdown table
- Adding vector.customConfig.sources.snowplow_bad_events.auto_offset_reset to markdown table
- Adding vector.customConfig.sources.snowplow_bad_events.group_id to markdown table
- Adding vector.customConfig.sources.snowplow_bad_events.topics to markdown table
- Adding vector.customConfig.sources.snowplow_bad_events.topics.0 to markdown table
- Adding vector.customConfig.sources.snowplow_bad_events.tls.enabled to markdown table
- Adding vector.customConfig.sources.snowplow_bad_events.tls.verify_certificate to markdown table
- Adding vector.customConfig.sources.snowplow_bad_events.tls.ca_file to markdown table
- Adding vector.customConfig.sources.snowplow_bad_events.tls.crt_file to markdown table
- Adding vector.customConfig.sources.snowplow_bad_events.tls.key_file to markdown table
- Adding vector.customConfig.sources.snowplow_bad_events.tls.key_pass to markdown table
- Adding vector.customConfig.sinks.clickhouse_enriched_events.type to markdown table
- Adding vector.customConfig.sinks.clickhouse_enriched_events.inputs to markdown table
- Adding vector.customConfig.sinks.clickhouse_enriched_events.inputs.0 to markdown table
- Adding vector.customConfig.sinks.clickhouse_enriched_events.endpoint to markdown table
- Adding vector.customConfig.sinks.clickhouse_enriched_events.database to markdown table
- Adding vector.customConfig.sinks.clickhouse_enriched_events.table to markdown table
- Adding vector.customConfig.sinks.clickhouse_enriched_events.skip_unknown_fields to markdown table
- Adding vector.customConfig.sinks.clickhouse_enriched_events.auth.strategy to markdown table
- Adding vector.customConfig.sinks.clickhouse_enriched_events.auth.user to markdown table
- Adding vector.customConfig.sinks.clickhouse_enriched_events.auth.password to markdown table
- Adding vector.customConfig.sinks.clickhouse_enriched_events.buffer.max_events to markdown table
- Adding vector.customConfig.sinks.clickhouse_enriched_events.batch.max_bytes to markdown table
- Adding vector.customConfig.sinks.clickhouse_enriched_events.batch.timeout_secs to markdown table
- Adding vector.customConfig.sinks.clickhouse_enriched_events.acknowledgements.enabled to markdown table
- Adding vector.customConfig.sinks.clickhouse_bad_events.type to markdown table
- Adding vector.customConfig.sinks.clickhouse_bad_events.inputs to markdown table
- Adding vector.customConfig.sinks.clickhouse_bad_events.inputs.0 to markdown table
- Adding vector.customConfig.sinks.clickhouse_bad_events.endpoint to markdown table
- Adding vector.customConfig.sinks.clickhouse_bad_events.database to markdown table
- Adding vector.customConfig.sinks.clickhouse_bad_events.table to markdown table
- Adding vector.customConfig.sinks.clickhouse_bad_events.skip_unknown_fields to markdown table
- Adding vector.customConfig.sinks.clickhouse_bad_events.auth.strategy to markdown table
- Adding vector.customConfig.sinks.clickhouse_bad_events.auth.user to markdown table
- Adding vector.customConfig.sinks.clickhouse_bad_events.auth.password to markdown table
- Adding vector.customConfig.sinks.clickhouse_bad_events.buffer.max_events to markdown table
- Adding vector.customConfig.sinks.clickhouse_bad_events.batch.max_bytes to markdown table
- Adding vector.customConfig.sinks.clickhouse_bad_events.batch.timeout_secs to markdown table
- Adding vector.customConfig.sinks.clickhouse_bad_events.acknowledgements.enabled to markdown table
- Adding vector.extraVolumeMounts to markdown table
- Adding vector.extraVolumeMounts.0.name to markdown table
- Adding vector.extraVolumeMounts.0.mountPath to markdown table
- Adding vector.persistence.enabled to markdown table
- Adding prometheus.alertmanager.enabled to markdown table
- Adding prometheus.prometheus-node-exporter.enabled to markdown table
- Adding prometheus.prometheus-pushgateway.enabled to markdown table
- Adding prometheus.server.remoteWrite to markdown table

docs/values-list.md successfully updated!

How to deploy upon merging

Numbered steps to explain how this change needs to be deployed. For instance, if there are any changes that should be made outside of the code changes themselves.

Related to https://gitlab.com/gitlab-org/analytics-section/product-analytics/analytics-stack/-/issues/160+

Edited by Robert Hunt

Merge request reports