Chef Failing on Prometheus Alert Managers
Chef is failing to run on alerts-01-inf-gprd.c.gitlab-production.internal
and alerts-02-inf-gprd.c.gitlab-production.internal
with the following error:
* template[/opt/prometheus/alertmanager/alertmanager.yml] action create[2018-09-18T23:32:58+00:00] INFO: Processing template[/opt/prometheus/alertmanager/alertmanager.yml] action create (gitlab-alertmanager::default line 70)
================================================================================
Error executing action `create` on resource 'template[/opt/prometheus/alertmanager/alertmanager.yml]'
================================================================================
Chef::Mixin::Template::TemplateError
------------------------------------
undefined method `[]' for nil:NilClass
Resource Declaration:
---------------------
# In /var/chef/cache/cookbooks/gitlab-alertmanager/recipes/default.rb
70: template node["alertmanager"]["flags"]["config.file"] do
71: source "alertmanager.yml.erb"
72: owner node["prometheus"]["user"]
73: group node["prometheus"]["group"]
74: mode "0644"
75: variables(conf: alertmanager_conf)
76: notifies :hup, "runit_service[alertmanager]"
77: end
78:
Compiled Resource:
------------------
# Declared in /var/chef/cache/cookbooks/gitlab-alertmanager/recipes/default.rb:70:in `from_file'
template("/opt/prometheus/alertmanager/alertmanager.yml") do
action [:create]
retries 0
retry_delay 2
default_guard_interpreter :default
source "alertmanager.yml.erb"
variables {:conf=>{"slack"=>{"channel"=>"#alerts", "api_url"=>"https://hooks.slack.com/services/XXXXX/XXXXX/XXXXX"}, "pagerduty"=>{"service_key"=>"XXXXX", "low_prio_service_key"=>"XXXXX"}}}
declared_type :template
cookbook_name "gitlab-alertmanager"
recipe_name "default"
owner "prometheus"
group "prometheus"
mode "0644"
path "/opt/prometheus/alertmanager/alertmanager.yml"
verifications []
end
Template Context:
-----------------
on line #225
223: - name: dead_mans_snitch
224: webhook_configs:
225: - url: "https://nosnch.in/<%= @conf['snitch']['api_key'] %>"
226: send_resolved: false
Platform:
---------
x86_64-linux
[2018-09-18T23:32:58+00:00] INFO: Running queued delayed notifications before re-raising exception
Running handlers:
[2018-09-18T23:32:58+00:00] ERROR: Running exception handlers
- PrometheusHandler
Running handlers complete
[2018-09-18T23:32:58+00:00] ERROR: Exception handlers complete
Chef Client failed. 14 resources updated in 31 seconds
[2018-09-18T23:32:58+00:00] FATAL: Stacktrace dumped to /var/chef/cache/chef-stacktrace.out
[2018-09-18T23:32:58+00:00] FATAL: Please provide the contents of the stacktrace.out file if you file a bug report
[2018-09-18T23:32:58+00:00] ERROR:
Chef::Mixin::Template::TemplateError (undefined method `[]' for nil:NilClass) on line #225:
223: - name: dead_mans_snitch
224: webhook_configs:
225: - url: "https://nosnch.in/<%= @conf['snitch']['api_key'] %>"
226: send_resolved: false
Edited by John Northrup