Infer diffable config files from config template tasks

What does this merge request do and why?

Infer diffable files from defined file tasks instead of hardcoding them in GDK::DIFFABLE_FILES as we tend to forget them if new config files are added.

This MR contains a couple of commits which prepare the crux of the MR: Infer diffable files.

1. It converts a couple of file <config> rake tasks to tasks.add_template_task definitions
   • For this, we added two config task options: rake_dependencies and generate_makefile_target
2. It moves config task definition from config.rake to lib/gdk/task_helpers/config_tasks.rb to make them reusable
3. Finally it removes GDK::DIFFABLE_FILES in favor of GDK::TaskHelpers::ConfigTasks.build.template_tasks
4. It excludes gitlab-pages-secret from being diffed via gdk diff-config by adding hide_diff

Please see individual commits for scoped context.

Contributes to #284 (closed)

How to set up and validate locally

1. rake all / rake reconfigure
   • Now more files are diffed, for example: grafana/grafana.ini - see the verification section below
2. rake --tasks | grep Generate
   • The amount of rake tasks remains the same
3. rake --trace all
   • The same tasks to generate configs are called
4. rake gdk.example.yml - no changes!

Verification grafana/grafana.ini

# On this branch
echo "# OHAI" >> support/templates/grafana/grafana.ini.erb
gdk diff-config
diff --git a/grafana/grafana.ini b/home/peter/devel/gitlab/gdk/tmp/diff_grafana_grafana.ini
index 82aa391b..88c7f655 100644
--- a/grafana/grafana.ini
+++ b/home/peter/devel/gitlab/gdk/tmp/diff_grafana_grafana.ini
@@ -2,3 +2,4 @@
 protocol = http
 http_addr = 10.23.0.5
 http_port = 4000
+# OHAI

rake all

ℹ️  'grafana/grafana.ini' has incoming changes:
-------------------------------------------------------------------------------------------------------------
@@ -2,3 +2,4 @@
 protocol = http
 http_addr = 10.23.0.5
 http_port = 4000
+# Hello

-------------------------------------------------------------------------------------------------------------
⚠️  WARNING: 'grafana/grafana.ini' has been overwritten. To recover the previous version, run:

cp -f '/home/peter/devel/gitlab/gdk/.backups/grafana__grafana.ini.20240711141053' \
'/home/peter/devel/gitlab/gdk/grafana/grafana.ini'

If you want to protect this file from being overwritten, see:
https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/main/doc/configuration.md#overwriting-configuration-files
-------------------------------------------------------------------------------------------------------------

# On main
echo "# OHAI" >> support/templates/grafana/grafana.ini.erb
# No output!
rake all
# No changes

Verification diffable files

# On this branch
bundle exec irb -Ilib -rgdk -rrake
File.write("after.tasks", GDK::TaskHelpers::ConfigTasks.build.template_tasks.map(&:name).sort.join("\n"))

# On main
bundle exec irb -Ilib -rgdk -rrake
File.write("before.tasks", GDK::DIFFABLE_FILES.sort.join("\n"))
diff -u before.tasks after.tasks

--- before.tasks	2024-07-11 13:49:32.264579627 +0200
+++ after.tasks	2024-07-11 13:48:57.341077617 +0200
@@ -8,12 +8,19 @@
 clickhouse/users.d/gdk.xml
 clickhouse/users.xml
 consul/config.json
+elasticsearch/config/elasticsearch.yml
+elasticsearch/config/jvm.options.d/custom.options
 gdk.example.yml
+gitaly/gitaly-0.praefect.toml
 gitaly/gitaly.config.toml
 gitaly/praefect.config.toml
+gitlab-k8s-agent-config.yml
+gitlab-pages-secret
 gitlab-pages/gitlab-pages.conf
 gitlab-runner-config.toml
 gitlab-shell/config.yml
+gitlab-spamcheck/config/config.toml
+gitlab-topology-service/config.toml
 gitlab/config/cable.yml
 gitlab/config/database.yml
 gitlab/config/gitlab.yml
@@ -29,6 +36,7 @@
 gitlab/config/session_store.yml
 gitlab/config/vite.gdk.json
 gitlab/workhorse/config.toml
+grafana/grafana.ini
 nginx/conf/nginx.conf
 openssh/sshd_config
 pgbouncers/pgbouncer-replica-1.ini
@@ -39,4 +47,6 @@
 prometheus/prometheus.yml
 redis/redis.conf
 registry/config.yml
+snowplow/iglu.json
+snowplow/snowplow_micro.conf
 support/makefiles/Makefile.config.mk
\ No newline at end of file

This means we are now able to diff more config files and show changes to the user.

Verification rake tasks

# On this branch
rake -T > after

# On main
rake -T > before

diff -u before after

--- before	2024-07-11 13:57:59.725392044 +0200
+++ after	2024-07-11 13:57:56.233441287 +0200
@@ -16,10 +16,11 @@
 rake dump_config                                        # Dump the configured settings
 rake elasticsearch/config/elasticsearch.yml             # Generate elasticsearch/config/elasticsearch.yml
 rake elasticsearch/config/jvm.options.d/custom.options  # Generate elasticsearch/config/jvm.options.d/custom.options
-rake gdk.example.yml                                    # Generate an example config file with all the defaults
+rake gdk.example.yml                                    # Generate gdk.example.yml
 rake git:configure[global]                              # Configure your Git with recommended settings
-rake gitaly/gitaly-0.praefect.toml                      # Generate gitaly config for praefect-internal-0
-rake gitaly/gitaly.config.toml                          # Generate gitaly config toml
+rake gitaly/gitaly-0.praefect.toml                      # Generate gitaly/gitaly-0.praefect.toml
+rake gitaly/gitaly.config.toml                          # Generate gitaly/gitaly.config.toml
+rake gitaly/praefect.config.toml                        # Generate gitaly/praefect.config.toml
 rake gitlab-db-migrate                                  # Run GitLab migrations
 rake gitlab-k8s-agent-config.yml                        # Generate gitlab-k8s-agent-config.yml
 rake gitlab-pages-secret                                # Generate gitlab-pages-secret
@@ -61,5 +62,5 @@
 rake registry/config.yml                                # Generate registry/config.yml
 rake snowplow/iglu.json                                 # Generate snowplow/iglu.json
 rake snowplow/snowplow_micro.conf                       # Generate snowplow/snowplow_micro.conf
-rake support/makefiles/Makefile.config.mk               # Dynamically generate Make targets for Rake tasks
+rake support/makefiles/Makefile.config.mk               # Generate support/makefiles/Makefile.config.mk
 rake vault:configure[project_id]                        # Vault configuration for a specific GitLab project

This means that only rake task description were changed and gitaly/gitaly-0.praefect.toml was added (which depends on praefect.node_count). In a follow-up we could pass add_template_task(name: x, desc: nil) for these type of tasks to hide them.

Verification config file generation

# On this branch
rake --trace=stdout all | sort > before.trace

# On main
rake --trace=stdout all | sort > after.trace
diff -u before.trace after.trace

--- before.trace	2024-07-11 14:03:12.204992666 +0200
+++ after.trace	2024-07-11 14:03:17.816913764 +0200
@@ -9,6 +9,7 @@
 ** Execute clickhouse/users.xml
 ** Execute clobber:gdk.example.yml
 ** Execute consul/config.json
+** Execute elasticsearch/config/jvm.options.d/custom.options
 ** Execute gdk.example.yml
 ** Execute gitaly/gitaly-0.praefect.toml
 ** Execute gitaly/gitaly.config.toml
@@ -27,10 +28,14 @@
 ** Execute gitlab/config/resque.yml
 ** Execute gitlab/config/session_store.yml
 ** Execute gitlab/config/vite.gdk.json
+** Execute gitlab-k8s-agent-config.yml
 ** Execute gitlab-pages/gitlab-pages.conf
 ** Execute gitlab-runner-config.toml
 ** Execute gitlab-shell/config.yml
+** Execute gitlab-spamcheck/config/config.toml
+** Execute gitlab-topology-service/config.toml
 ** Execute gitlab/workhorse/config.toml
+** Execute grafana/grafana.ini
 ** Execute nginx/conf/nginx.conf
 ** Execute openssh/sshd_config
 ** Execute pgbouncers/pgbouncer-replica-1.ini
@@ -42,6 +47,8 @@
 ** Execute prometheus/prometheus.yml
 ** Execute redis/redis.conf
 ** Execute registry/config.yml
+** Execute snowplow/iglu.json
+** Execute snowplow/snowplow_micro.conf
 ** Invoke all (first_time)
 ** Invoke clickhouse/config.d/data-paths.xml (first_time)
 ** Invoke clickhouse/config.d/gdk.xml (first_time)
@@ -53,7 +60,10 @@
 ** Invoke clickhouse/users.xml (first_time)
 ** Invoke clobber:gdk.example.yml (first_time)
 ** Invoke consul/config.json (first_time)
+** Invoke elasticsearch/config/elasticsearch.yml (first_time, not_needed)
+** Invoke elasticsearch/config/jvm.options.d/custom.options (first_time)
 ** Invoke gdk.example.yml (first_time)
+** Invoke gitaly/gitaly-0.praefect.toml 
 ** Invoke gitaly/gitaly-0.praefect.toml (first_time)
 ** Invoke gitaly/gitaly.config.toml (first_time)
 ** Invoke gitaly/praefect.config.toml (first_time)
@@ -71,10 +81,15 @@
 ** Invoke gitlab/config/resque.yml (first_time)
 ** Invoke gitlab/config/session_store.yml (first_time)
 ** Invoke gitlab/config/vite.gdk.json (first_time)
+** Invoke gitlab-k8s-agent-config.yml (first_time)
 ** Invoke gitlab-pages/gitlab-pages.conf (first_time)
+** Invoke gitlab-pages-secret (first_time, not_needed)
 ** Invoke gitlab-runner-config.toml (first_time)
 ** Invoke gitlab-shell/config.yml (first_time)
+** Invoke gitlab-spamcheck/config/config.toml (first_time)
+** Invoke gitlab-topology-service/config.toml (first_time)
 ** Invoke gitlab/workhorse/config.toml (first_time)
+** Invoke grafana/grafana.ini (first_time)
 ** Invoke /home/peter/devel/gitlab/gdk/gdk.yml (first_time, not_needed)
 ** Invoke /home/peter/devel/gitlab/gdk/gdk.yml (not_needed)
 ** Invoke /home/peter/devel/gitlab/gdk/gdk.yml (not_needed)
@@ -113,6 +128,20 @@
 ** Invoke /home/peter/devel/gitlab/gdk/gdk.yml (not_needed)
 ** Invoke /home/peter/devel/gitlab/gdk/gdk.yml (not_needed)
 ** Invoke /home/peter/devel/gitlab/gdk/gdk.yml (not_needed)
+** Invoke /home/peter/devel/gitlab/gdk/gdk.yml (not_needed)
+** Invoke /home/peter/devel/gitlab/gdk/gdk.yml (not_needed)
+** Invoke /home/peter/devel/gitlab/gdk/gdk.yml (not_needed)
+** Invoke /home/peter/devel/gitlab/gdk/gdk.yml (not_needed)
+** Invoke /home/peter/devel/gitlab/gdk/gdk.yml (not_needed)
+** Invoke /home/peter/devel/gitlab/gdk/gdk.yml (not_needed)
+** Invoke /home/peter/devel/gitlab/gdk/gdk.yml (not_needed)
+** Invoke /home/peter/devel/gitlab/gdk/gdk.yml (not_needed)
+** Invoke /home/peter/devel/gitlab/gdk/gdk.yml (not_needed)
+** Invoke /home/peter/devel/gitlab/gdk/gdk.yml (not_needed)
+** Invoke /home/peter/devel/gitlab/gdk/gdk.yml (not_needed)
+** Invoke /home/peter/devel/gitlab/gdk/gdk.yml (not_needed)
+** Invoke /home/peter/devel/gitlab/gdk/gdk.yml (not_needed)
+** Invoke /home/peter/devel/gitlab/gdk/gdk.yml (not_needed)
 ** Invoke lib/asdf (first_time, not_needed)
 ** Invoke lib/asdf.rb (first_time, not_needed)
 ** Invoke lib/asdf/tool.rb (first_time, not_needed)
@@ -289,6 +318,8 @@
 ** Invoke prometheus/prometheus.yml (first_time)
 ** Invoke redis/redis.conf (first_time)
 ** Invoke registry/config.yml (first_time)
+** Invoke snowplow/iglu.json (first_time)
+** Invoke snowplow/snowplow_micro.conf (first_time)
 ** Invoke support/makefiles/Makefile.config.mk (first_time, not_needed)
 ** Invoke support/templates/clickhouse/config.d/data-paths.xml.erb (first_time, not_needed)
 ** Invoke support/templates/clickhouse/config.d/gdk.xml.erb (first_time, not_needed)
@@ -299,6 +330,9 @@
 ** Invoke support/templates/clickhouse/users.d/gdk.xml.erb (first_time, not_needed)
 ** Invoke support/templates/clickhouse/users.xml (first_time, not_needed)
 ** Invoke support/templates/consul/config.json.erb (first_time, not_needed)
+** Invoke support/templates/elasticsearch/config/elasticsearch.yml (first_time, not_needed)
+** Invoke support/templates/elasticsearch/config/jvm.options.d/custom.options (first_time, not_needed)
+** Invoke support/templates/gdk.example.yml.erb (first_time, not_needed)
 ** Invoke support/templates/gitaly/gitaly.config.toml.erb (first_time, not_needed)
 ** Invoke support/templates/gitaly/gitaly.config.toml.erb (not_needed)
 ** Invoke support/templates/gitaly/praefect.config.toml.erb (first_time, not_needed)
@@ -316,10 +350,16 @@
 ** Invoke support/templates/gitlab/config/redis.sessions.yml.erb (not_needed)
 ** Invoke support/templates/gitlab/config/session_store.yml.erb (first_time, not_needed)
 ** Invoke support/templates/gitlab/config/vite.gdk.json.erb (first_time, not_needed)
+** Invoke support/templates/gitlab-k8s-agent-config.yml.erb (first_time, not_needed)
 ** Invoke support/templates/gitlab-pages/gitlab-pages.conf.erb (first_time, not_needed)
+** Invoke support/templates/gitlab-pages-secret.erb (first_time, not_needed)
 ** Invoke support/templates/gitlab-runner-config.toml.erb (first_time, not_needed)
 ** Invoke support/templates/gitlab-shell/config.yml.erb (first_time, not_needed)
+** Invoke support/templates/gitlab-spamcheck/config/config.toml.erb (first_time, not_needed)
+** Invoke support/templates/gitlab-topology-service/config.toml.erb (first_time, not_needed)
 ** Invoke support/templates/gitlab/workhorse/config.toml.erb (first_time, not_needed)
+** Invoke support/templates/grafana/grafana.ini.erb (first_time, not_needed)
+** Invoke support/templates/makefiles/Makefile.config.mk.erb (first_time, not_needed)
 ** Invoke support/templates/nginx/conf/nginx.conf.erb (first_time, not_needed)
 ** Invoke support/templates/openssh/sshd_config.erb (first_time, not_needed)
 ** Invoke support/templates/pgbouncer/pgbouncer-replica.ini.erb (first_time, not_needed)
@@ -331,4 +371,6 @@
 ** Invoke support/templates/prometheus/prometheus.yml.erb (first_time, not_needed)
 ** Invoke support/templates/redis/redis.conf.erb (first_time, not_needed)
 ** Invoke support/templates/registry/config.yml.erb (first_time, not_needed)
-rm -r clobber:gdk.example.yml
+** Invoke support/templates/snowplow/iglu.json.erb (first_time, not_needed)
+** Invoke support/templates/snowplow/snowplow_micro.conf.erb (first_time, not_needed)
+rm -r gdk.example.yml

This confirms that we are not generating more files - as expected.

Impacted categories

The following categories relate to this merge request:

• gdk-reliability - e.g. When a GDK action fails to complete.
• gdk-usability - e.g. Improvements or suggestions around how the GDK functions.
• gdk-performance - e.g. When a GDK action is slow or times out.

Merge request checklist

• This change is backward compatible. If not, please include steps to communicate to our users.
• Tests added for new functionality. If not, please raise an issue to follow-up.
• Documentation added/updated, if needed.
• Announcement added, if change is notable.
• gdk doctor test added, if needed.
• Add the ~highlight label if this MR should be included in the CHANGELOG.md.