Skip to content
Snippets Groups Projects

gitaly-cluster

Passed Started by
🤖 GitLab Bot 🤖
@gitlab-bot
Showing last 499.89 KiB of log. View raw or full log.
1[2023-07-20T03:20:22+00:00] INFO: directory[/var/log/gitlab/logrotate] mode changed to 700
2 - change mode from '' to '0700'
3 - change owner from '' to 'root'
4 * template[/var/opt/gitlab/logrotate/logrotate.conf] action create[2023-07-20T03:20:22+00:00] INFO: template[/var/opt/gitlab/logrotate/logrotate.conf] created file /var/opt/gitlab/logrotate/logrotate.conf
5 - create new file /var/opt/gitlab/logrotate/logrotate.conf[2023-07-20T03:20:22+00:00] INFO: template[/var/opt/gitlab/logrotate/logrotate.conf] updated file contents /var/opt/gitlab/logrotate/logrotate.conf
6 - update content in file /var/opt/gitlab/logrotate/logrotate.conf from none to c1c320
7 --- /var/opt/gitlab/logrotate/logrotate.conf 2023-07-20 03:20:22.000000000 +0000
8 +++ /var/opt/gitlab/logrotate/.chef-logrotate20230720-37-c8heoi.conf 2023-07-20 03:20:22.000000000 +0000
9 @@ -1,12 +1,24 @@
10 +# Generated by 'gitlab-ctl reconfigure'.
11 +# Modifications will be overwritten!
12 +
13 +include /var/opt/gitlab/logrotate/logrotate.d/nginx
14 +include /var/opt/gitlab/logrotate/logrotate.d/puma
15 +include /var/opt/gitlab/logrotate/logrotate.d/gitlab-rails
16 +include /var/opt/gitlab/logrotate/logrotate.d/gitlab-shell
17 +include /var/opt/gitlab/logrotate/logrotate.d/gitlab-workhorse
18 +include /var/opt/gitlab/logrotate/logrotate.d/gitlab-pages
19 +include /var/opt/gitlab/logrotate/logrotate.d/gitlab-kas
20 +include /var/opt/gitlab/logrotate/logrotate.d/gitaly
21 +include /var/opt/gitlab/logrotate/logrotate.d/mailroom[2023-07-20T03:20:22+00:00] INFO: template[/var/opt/gitlab/logrotate/logrotate.conf] mode changed to 644
22 - change mode from '' to '0644'
23 * template[/var/opt/gitlab/logrotate/logrotate.d/nginx] action create[2023-07-20T03:20:22+00:00] INFO: template[/var/opt/gitlab/logrotate/logrotate.d/nginx] created file /var/opt/gitlab/logrotate/logrotate.d/nginx
24 - create new file /var/opt/gitlab/logrotate/logrotate.d/nginx[2023-07-20T03:20:22+00:00] INFO: template[/var/opt/gitlab/logrotate/logrotate.d/nginx] updated file contents /var/opt/gitlab/logrotate/logrotate.d/nginx
25 - update content in file /var/opt/gitlab/logrotate/logrotate.d/nginx from none to e17e19
26 --- /var/opt/gitlab/logrotate/logrotate.d/nginx 2023-07-20 03:20:22.000000000 +0000
27 +++ /var/opt/gitlab/logrotate/logrotate.d/.chef-nginx20230720-37-qehqzc 2023-07-20 03:20:22.000000000 +0000
28 @@ -1,17 +1,34 @@
29 +# Generated by gitlab-ctl reconfigure
30 +# Modifications will be overwritten!
31 +
32 +/var/log/gitlab/nginx/*.log {
33 + su root root
34 + daily
35 +
36 +
37 + rotate 30
38 + compress
39 + copytruncate
40 + missingok
41 + notifempty
42 + postrotate
43 +
44 + endscript
45 +}
46 * template[/var/opt/gitlab/logrotate/logrotate.d/puma] action create[2023-07-20T03:20:22+00:00] INFO: template[/var/opt/gitlab/logrotate/logrotate.d/puma] created file /var/opt/gitlab/logrotate/logrotate.d/puma
47 - create new file /var/opt/gitlab/logrotate/logrotate.d/puma[2023-07-20T03:20:22+00:00] INFO: template[/var/opt/gitlab/logrotate/logrotate.d/puma] updated file contents /var/opt/gitlab/logrotate/logrotate.d/puma
48 - update content in file /var/opt/gitlab/logrotate/logrotate.d/puma from none to 44ad27
49 --- /var/opt/gitlab/logrotate/logrotate.d/puma 2023-07-20 03:20:22.000000000 +0000
50 +++ /var/opt/gitlab/logrotate/logrotate.d/.chef-puma20230720-37-o1sh6r 2023-07-20 03:20:22.000000000 +0000
51 @@ -1,17 +1,34 @@
52 +# Generated by gitlab-ctl reconfigure
53 +# Modifications will be overwritten!
54 +
55 +/var/log/gitlab/puma/*.log {
56 + su git git
57 + daily
58 +
59 +
60 + rotate 30
61 + compress
62 + copytruncate
63 + missingok
64 + notifempty
65 + postrotate
66 +
67 + endscript
68 +}
69 * template[/var/opt/gitlab/logrotate/logrotate.d/gitlab-rails] action create[2023-07-20T03:20:22+00:00] INFO: template[/var/opt/gitlab/logrotate/logrotate.d/gitlab-rails] created file /var/opt/gitlab/logrotate/logrotate.d/gitlab-rails
70 - create new file /var/opt/gitlab/logrotate/logrotate.d/gitlab-rails[2023-07-20T03:20:22+00:00] INFO: template[/var/opt/gitlab/logrotate/logrotate.d/gitlab-rails] updated file contents /var/opt/gitlab/logrotate/logrotate.d/gitlab-rails
71 - update content in file /var/opt/gitlab/logrotate/logrotate.d/gitlab-rails from none to 26254a
72 --- /var/opt/gitlab/logrotate/logrotate.d/gitlab-rails 2023-07-20 03:20:22.000000000 +0000
73 +++ /var/opt/gitlab/logrotate/logrotate.d/.chef-gitlab-rails20230720-37-34mq7k 2023-07-20 03:20:22.000000000 +0000
74 @@ -1,17 +1,34 @@
75 +# Generated by gitlab-ctl reconfigure
76 +# Modifications will be overwritten!
77 +
78 +/var/log/gitlab/gitlab-rails/*.log {
79 + su git git
80 + daily
81 +
82 +
83 + rotate 30
84 + compress
85 + copytruncate
86 + missingok
87 + notifempty
88 + postrotate
89 +
90 + endscript
91 +}
92 * template[/var/opt/gitlab/logrotate/logrotate.d/gitlab-shell] action create[2023-07-20T03:20:22+00:00] INFO: template[/var/opt/gitlab/logrotate/logrotate.d/gitlab-shell] created file /var/opt/gitlab/logrotate/logrotate.d/gitlab-shell
93 - create new file /var/opt/gitlab/logrotate/logrotate.d/gitlab-shell[2023-07-20T03:20:22+00:00] INFO: template[/var/opt/gitlab/logrotate/logrotate.d/gitlab-shell] updated file contents /var/opt/gitlab/logrotate/logrotate.d/gitlab-shell
94 - update content in file /var/opt/gitlab/logrotate/logrotate.d/gitlab-shell from none to 859f56
95 --- /var/opt/gitlab/logrotate/logrotate.d/gitlab-shell 2023-07-20 03:20:22.000000000 +0000
96 +++ /var/opt/gitlab/logrotate/logrotate.d/.chef-gitlab-shell20230720-37-gdsqzm 2023-07-20 03:20:22.000000000 +0000
97 @@ -1,17 +1,34 @@
98 +# Generated by gitlab-ctl reconfigure
99 +# Modifications will be overwritten!
100 +
101 +/var/log/gitlab/gitlab-shell/*.log {
102 + su git git
103 + daily
104 +
105 +
106 + rotate 30
107 + compress
108 + copytruncate
109 + missingok
110 + notifempty
111 + postrotate
112 +
113 + endscript
114 +}
115 * template[/var/opt/gitlab/logrotate/logrotate.d/gitlab-workhorse] action create[2023-07-20T03:20:22+00:00] INFO: template[/var/opt/gitlab/logrotate/logrotate.d/gitlab-workhorse] created file /var/opt/gitlab/logrotate/logrotate.d/gitlab-workhorse
116 - create new file /var/opt/gitlab/logrotate/logrotate.d/gitlab-workhorse[2023-07-20T03:20:22+00:00] INFO: template[/var/opt/gitlab/logrotate/logrotate.d/gitlab-workhorse] updated file contents /var/opt/gitlab/logrotate/logrotate.d/gitlab-workhorse
117 - update content in file /var/opt/gitlab/logrotate/logrotate.d/gitlab-workhorse from none to 3453f6
118 --- /var/opt/gitlab/logrotate/logrotate.d/gitlab-workhorse 2023-07-20 03:20:22.000000000 +0000
119 +++ /var/opt/gitlab/logrotate/logrotate.d/.chef-gitlab-workhorse20230720-37-2unx1g 2023-07-20 03:20:22.000000000 +0000
120 @@ -1,17 +1,34 @@
121 +# Generated by gitlab-ctl reconfigure
122 +# Modifications will be overwritten!
123 +
124 +/var/log/gitlab/gitlab-workhorse/*.log {
125 + su git git
126 + daily
127 +
128 +
129 + rotate 30
130 + compress
131 + copytruncate
132 + missingok
133 + notifempty
134 + postrotate
135 +
136 + endscript
137 +}
138 * template[/var/opt/gitlab/logrotate/logrotate.d/gitlab-pages] action create[2023-07-20T03:20:22+00:00] INFO: template[/var/opt/gitlab/logrotate/logrotate.d/gitlab-pages] created file /var/opt/gitlab/logrotate/logrotate.d/gitlab-pages
139 - create new file /var/opt/gitlab/logrotate/logrotate.d/gitlab-pages[2023-07-20T03:20:22+00:00] INFO: template[/var/opt/gitlab/logrotate/logrotate.d/gitlab-pages] updated file contents /var/opt/gitlab/logrotate/logrotate.d/gitlab-pages
140 - update content in file /var/opt/gitlab/logrotate/logrotate.d/gitlab-pages from none to 3c118a
141 --- /var/opt/gitlab/logrotate/logrotate.d/gitlab-pages 2023-07-20 03:20:22.000000000 +0000
142 +++ /var/opt/gitlab/logrotate/logrotate.d/.chef-gitlab-pages20230720-37-rkj96d 2023-07-20 03:20:22.000000000 +0000
143 @@ -1,17 +1,34 @@
144 +# Generated by gitlab-ctl reconfigure
145 +# Modifications will be overwritten!
146 +
147 +/var/log/gitlab/gitlab-pages/*.log {
148 + su git git
149 + daily
150 +
151 +
152 + rotate 30
153 + compress
154 + copytruncate
155 + missingok
156 + notifempty
157 + postrotate
158 +
159 + endscript
160 +}
161 * template[/var/opt/gitlab/logrotate/logrotate.d/gitlab-kas] action create[2023-07-20T03:20:22+00:00] INFO: template[/var/opt/gitlab/logrotate/logrotate.d/gitlab-kas] created file /var/opt/gitlab/logrotate/logrotate.d/gitlab-kas
162 - create new file /var/opt/gitlab/logrotate/logrotate.d/gitlab-kas[2023-07-20T03:20:22+00:00] INFO: template[/var/opt/gitlab/logrotate/logrotate.d/gitlab-kas] updated file contents /var/opt/gitlab/logrotate/logrotate.d/gitlab-kas
163 - update content in file /var/opt/gitlab/logrotate/logrotate.d/gitlab-kas from none to 0ba4f0
164 --- /var/opt/gitlab/logrotate/logrotate.d/gitlab-kas 2023-07-20 03:20:22.000000000 +0000
165 +++ /var/opt/gitlab/logrotate/logrotate.d/.chef-gitlab-kas20230720-37-nsuqov 2023-07-20 03:20:22.000000000 +0000
166 @@ -1,17 +1,34 @@
167 +# Generated by gitlab-ctl reconfigure
168 +# Modifications will be overwritten!
169 +
170 +/var/log/gitlab/gitlab-kas/*.log {
171 + su git git
172 + daily
173 +
174 +
175 + rotate 30
176 + compress
177 + copytruncate
178 + missingok
179 + notifempty
180 + postrotate
181 +
182 + endscript
183 +}
184 * template[/var/opt/gitlab/logrotate/logrotate.d/gitaly] action create[2023-07-20T03:20:22+00:00] INFO: template[/var/opt/gitlab/logrotate/logrotate.d/gitaly] created file /var/opt/gitlab/logrotate/logrotate.d/gitaly
185 - create new file /var/opt/gitlab/logrotate/logrotate.d/gitaly[2023-07-20T03:20:22+00:00] INFO: template[/var/opt/gitlab/logrotate/logrotate.d/gitaly] updated file contents /var/opt/gitlab/logrotate/logrotate.d/gitaly
186 - update content in file /var/opt/gitlab/logrotate/logrotate.d/gitaly from none to 02bda7
187 --- /var/opt/gitlab/logrotate/logrotate.d/gitaly 2023-07-20 03:20:22.000000000 +0000
188 +++ /var/opt/gitlab/logrotate/logrotate.d/.chef-gitaly20230720-37-7iculk 2023-07-20 03:20:22.000000000 +0000
189 @@ -1,17 +1,34 @@
190 +# Generated by gitlab-ctl reconfigure
191 +# Modifications will be overwritten!
192 +
193 +/var/log/gitlab/gitaly/*.log {
194 + su git git
195 + daily
196 +
197 +
198 + rotate 30
199 + compress
200 + copytruncate
201 + missingok
202 + notifempty
203 + postrotate
204 +
205 + endscript
206 +}
207 * template[/var/opt/gitlab/logrotate/logrotate.d/mailroom] action create[2023-07-20T03:20:22+00:00] INFO: template[/var/opt/gitlab/logrotate/logrotate.d/mailroom] created file /var/opt/gitlab/logrotate/logrotate.d/mailroom
208 - create new file /var/opt/gitlab/logrotate/logrotate.d/mailroom[2023-07-20T03:20:22+00:00] INFO: template[/var/opt/gitlab/logrotate/logrotate.d/mailroom] updated file contents /var/opt/gitlab/logrotate/logrotate.d/mailroom
209 - update content in file /var/opt/gitlab/logrotate/logrotate.d/mailroom from none to 8fcea3
210 --- /var/opt/gitlab/logrotate/logrotate.d/mailroom 2023-07-20 03:20:22.000000000 +0000
211 +++ /var/opt/gitlab/logrotate/logrotate.d/.chef-mailroom20230720-37-w54w2u 2023-07-20 03:20:22.000000000 +0000
212 @@ -1,17 +1,34 @@
213 +# Generated by gitlab-ctl reconfigure
214 +# Modifications will be overwritten!
215 +
216 +/var/log/gitlab/mailroom/*.log {
217 + su git git
218 + daily
219 +
220 +
221 + rotate 30
222 + compress
223 + copytruncate
224 + missingok
225 + notifempty
226 + postrotate
227 +
228 + endscript
229 +}
230Recipe: logrotate::enable
231 * service[logrotate] action nothing (skipped due to action :nothing)
232 * runit_service[logrotate] action enable
233 * ruby_block[restart_service] action nothing (skipped due to action :nothing)
234 * ruby_block[restart_log_service] action nothing (skipped due to action :nothing)
235 * ruby_block[reload_log_service] action nothing (skipped due to action :nothing)
236 * directory[/opt/gitlab/sv/logrotate] action create[2023-07-20T03:20:22+00:00] INFO: directory[/opt/gitlab/sv/logrotate] created directory /opt/gitlab/sv/logrotate
237 - create new directory /opt/gitlab/sv/logrotate[2023-07-20T03:20:22+00:00] INFO: directory[/opt/gitlab/sv/logrotate] owner changed to 0
238[2023-07-20T03:20:22+00:00] INFO: directory[/opt/gitlab/sv/logrotate] group changed to 0
239[2023-07-20T03:20:22+00:00] INFO: directory[/opt/gitlab/sv/logrotate] mode changed to 755
240 - change mode from '' to '0755'
241 - change owner from '' to 'root'
242 - change group from '' to 'root'
243 * template[/opt/gitlab/sv/logrotate/run] action create[2023-07-20T03:20:23+00:00] INFO: template[/opt/gitlab/sv/logrotate/run] created file /opt/gitlab/sv/logrotate/run
244 - create new file /opt/gitlab/sv/logrotate/run[2023-07-20T03:20:23+00:00] INFO: template[/opt/gitlab/sv/logrotate/run] updated file contents /opt/gitlab/sv/logrotate/run
245 - update content in file /opt/gitlab/sv/logrotate/run from none to 07f1b6
246 --- /opt/gitlab/sv/logrotate/run 2023-07-20 03:20:23.000000000 +0000
247 +++ /opt/gitlab/sv/logrotate/.chef-run20230720-37-x4do4w 2023-07-20 03:20:23.000000000 +0000
248 @@ -1,10 +1,20 @@
249 +#!/bin/sh
250 +exec 2>&1
251 +
252 +cd /var/opt/gitlab/logrotate
253 +
254 +exec /opt/gitlab/embedded/bin/chpst -P /usr/bin/env \
255 + dir=/var/opt/gitlab/logrotate \
256 + pre_sleep=600 \
257 + post_sleep=3000 \
258 + /opt/gitlab/embedded/bin/gitlab-logrotate-wrapper[2023-07-20T03:20:23+00:00] INFO: template[/opt/gitlab/sv/logrotate/run] owner changed to 0
259[2023-07-20T03:20:23+00:00] INFO: template[/opt/gitlab/sv/logrotate/run] group changed to 0
260[2023-07-20T03:20:23+00:00] INFO: template[/opt/gitlab/sv/logrotate/run] mode changed to 755
261 - change mode from '' to '0755'
262 - change owner from '' to 'root'
263 - change group from '' to 'root'
264 * directory[/opt/gitlab/sv/logrotate/log] action create[2023-07-20T03:20:23+00:00] INFO: directory[/opt/gitlab/sv/logrotate/log] created directory /opt/gitlab/sv/logrotate/log
265 - create new directory /opt/gitlab/sv/logrotate/log[2023-07-20T03:20:23+00:00] INFO: directory[/opt/gitlab/sv/logrotate/log] owner changed to 0
266[2023-07-20T03:20:23+00:00] INFO: directory[/opt/gitlab/sv/logrotate/log] group changed to 0
267[2023-07-20T03:20:23+00:00] INFO: directory[/opt/gitlab/sv/logrotate/log] mode changed to 755
268 - change mode from '' to '0755'
269 - change owner from '' to 'root'
270 - change group from '' to 'root'
271 * directory[/opt/gitlab/sv/logrotate/log/main] action create[2023-07-20T03:20:23+00:00] INFO: directory[/opt/gitlab/sv/logrotate/log/main] created directory /opt/gitlab/sv/logrotate/log/main
272 - create new directory /opt/gitlab/sv/logrotate/log/main[2023-07-20T03:20:23+00:00] INFO: directory[/opt/gitlab/sv/logrotate/log/main] owner changed to 0
273[2023-07-20T03:20:23+00:00] INFO: directory[/opt/gitlab/sv/logrotate/log/main] group changed to 0
274[2023-07-20T03:20:23+00:00] INFO: directory[/opt/gitlab/sv/logrotate/log/main] mode changed to 755
275 - change mode from '' to '0755'
276 - change owner from '' to 'root'
277 - change group from '' to 'root'
278 * template[/opt/gitlab/sv/logrotate/log/config] action create[2023-07-20T03:20:23+00:00] INFO: template[/opt/gitlab/sv/logrotate/log/config] created file /opt/gitlab/sv/logrotate/log/config
279 - create new file /opt/gitlab/sv/logrotate/log/config[2023-07-20T03:20:23+00:00] INFO: template[/opt/gitlab/sv/logrotate/log/config] updated file contents /opt/gitlab/sv/logrotate/log/config
280 - update content in file /opt/gitlab/sv/logrotate/log/config from none to 623c00
281 --- /opt/gitlab/sv/logrotate/log/config 2023-07-20 03:20:23.000000000 +0000
282 +++ /opt/gitlab/sv/logrotate/log/.chef-config20230720-37-rdb2rr 2023-07-20 03:20:23.000000000 +0000
283 @@ -1,6 +1,12 @@
284 +s209715200
285 +n30
286 +t86400
287 +!gzip
288 +
289 +[2023-07-20T03:20:23+00:00] INFO: template[/opt/gitlab/sv/logrotate/log/config] owner changed to 0
290[2023-07-20T03:20:23+00:00] INFO: template[/opt/gitlab/sv/logrotate/log/config] group changed to 0
291[2023-07-20T03:20:23+00:00] INFO: template[/opt/gitlab/sv/logrotate/log/config] mode changed to 644
292 - change mode from '' to '0644'
293 - change owner from '' to 'root'
294 - change group from '' to 'root'
295[2023-07-20T03:20:23+00:00] INFO: template[/opt/gitlab/sv/logrotate/log/config] sending create action to ruby_block[verify_chown_persisted_on_logrotate] (immediate)
296 * ruby_block[verify_chown_persisted_on_logrotate] action create[2023-07-20T03:20:23+00:00] INFO: ruby_block[verify_chown_persisted_on_logrotate] called
297 - execute the ruby block verify_chown_persisted_on_logrotate
298 * ruby_block[verify_chown_persisted_on_logrotate] action nothing (skipped due to action :nothing)
299 * link[/var/log/gitlab/logrotate/config] action create[2023-07-20T03:20:23+00:00] INFO: link[/var/log/gitlab/logrotate/config] created
300 - create symlink at /var/log/gitlab/logrotate/config to /opt/gitlab/sv/logrotate/log/config
301 * template[/opt/gitlab/sv/logrotate/log/run] action create[2023-07-20T03:20:23+00:00] INFO: template[/opt/gitlab/sv/logrotate/log/run] created file /opt/gitlab/sv/logrotate/log/run
302 - create new file /opt/gitlab/sv/logrotate/log/run[2023-07-20T03:20:23+00:00] INFO: template[/opt/gitlab/sv/logrotate/log/run] updated file contents /opt/gitlab/sv/logrotate/log/run
303 - update content in file /opt/gitlab/sv/logrotate/log/run from none to 637335
304 --- /opt/gitlab/sv/logrotate/log/run 2023-07-20 03:20:23.000000000 +0000
305 +++ /opt/gitlab/sv/logrotate/log/.chef-run20230720-37-pzyhey 2023-07-20 03:20:23.000000000 +0000
306 @@ -1,5 +1,10 @@
307 +#!/bin/sh
308 +exec chpst -P \
309 + -U root:root \
310 + -u root:root \
311 + svlogd -tt /var/log/gitlab/logrotate[2023-07-20T03:20:23+00:00] INFO: template[/opt/gitlab/sv/logrotate/log/run] owner changed to 0
312[2023-07-20T03:20:23+00:00] INFO: template[/opt/gitlab/sv/logrotate/log/run] group changed to 0
313[2023-07-20T03:20:23+00:00] INFO: template[/opt/gitlab/sv/logrotate/log/run] mode changed to 755
314 - change mode from '' to '0755'
315 - change owner from '' to 'root'
316 - change group from '' to 'root'
317 * directory[/opt/gitlab/sv/logrotate/env] action create[2023-07-20T03:20:23+00:00] INFO: directory[/opt/gitlab/sv/logrotate/env] created directory /opt/gitlab/sv/logrotate/env
318 - create new directory /opt/gitlab/sv/logrotate/env[2023-07-20T03:20:23+00:00] INFO: directory[/opt/gitlab/sv/logrotate/env] owner changed to 0
319[2023-07-20T03:20:23+00:00] INFO: directory[/opt/gitlab/sv/logrotate/env] group changed to 0
320[2023-07-20T03:20:23+00:00] INFO: directory[/opt/gitlab/sv/logrotate/env] mode changed to 755
321 - change mode from '' to '0755'
322 - change owner from '' to 'root'
323 - change group from '' to 'root'
324 * ruby_block[Delete unmanaged env files for logrotate service] action run (skipped due to only_if)
325 * template[/opt/gitlab/sv/logrotate/check] action create (skipped due to only_if)
326 * template[/opt/gitlab/sv/logrotate/finish] action create (skipped due to only_if)
327 * directory[/opt/gitlab/sv/logrotate/control] action create[2023-07-20T03:20:23+00:00] INFO: directory[/opt/gitlab/sv/logrotate/control] created directory /opt/gitlab/sv/logrotate/control
328 - create new directory /opt/gitlab/sv/logrotate/control[2023-07-20T03:20:23+00:00] INFO: directory[/opt/gitlab/sv/logrotate/control] owner changed to 0
329[2023-07-20T03:20:23+00:00] INFO: directory[/opt/gitlab/sv/logrotate/control] group changed to 0
330[2023-07-20T03:20:23+00:00] INFO: directory[/opt/gitlab/sv/logrotate/control] mode changed to 755
331 - change mode from '' to '0755'
332 - change owner from '' to 'root'
333 - change group from '' to 'root'
334 * template[/opt/gitlab/sv/logrotate/control/t] action create[2023-07-20T03:20:23+00:00] INFO: template[/opt/gitlab/sv/logrotate/control/t] created file /opt/gitlab/sv/logrotate/control/t
335 - create new file /opt/gitlab/sv/logrotate/control/t[2023-07-20T03:20:23+00:00] INFO: template[/opt/gitlab/sv/logrotate/control/t] updated file contents /opt/gitlab/sv/logrotate/control/t
336 - update content in file /opt/gitlab/sv/logrotate/control/t from none to 8fa3fa
337 --- /opt/gitlab/sv/logrotate/control/t 2023-07-20 03:20:23.000000000 +0000
338 +++ /opt/gitlab/sv/logrotate/control/.chef-t20230720-37-ikq73q 2023-07-20 03:20:23.000000000 +0000
339 @@ -1,3 +1,6 @@
340 +#!/bin/sh
341 +echo "Received TERM from runit, sending to process group (-PID)"
342 +kill -- -$(cat /opt/gitlab/service/logrotate/supervise/pid)[2023-07-20T03:20:23+00:00] INFO: template[/opt/gitlab/sv/logrotate/control/t] owner changed to 0
343[2023-07-20T03:20:23+00:00] INFO: template[/opt/gitlab/sv/logrotate/control/t] group changed to 0
344[2023-07-20T03:20:23+00:00] INFO: template[/opt/gitlab/sv/logrotate/control/t] mode changed to 755
345 - change mode from '' to '0755'
346 - change owner from '' to 'root'
347 - change group from '' to 'root'
348 * link[/opt/gitlab/init/logrotate] action create[2023-07-20T03:20:23+00:00] INFO: link[/opt/gitlab/init/logrotate] created
349 - create symlink at /opt/gitlab/init/logrotate to /opt/gitlab/embedded/bin/sv
350 * file[/opt/gitlab/sv/logrotate/down] action nothing (skipped due to action :nothing)
351[2023-07-20T03:20:23+00:00] INFO: template[/opt/gitlab/sv/logrotate/run] sending run action to ruby_block[restart_service] (delayed)
352 * ruby_block[restart_service] action run (skipped due to only_if)
353[2023-07-20T03:20:23+00:00] INFO: template[/opt/gitlab/sv/logrotate/log/config] sending create action to ruby_block[reload_log_service] (delayed)
354 * ruby_block[reload_log_service] action create
355 * ruby_block[restart_service] action nothing (skipped due to action :nothing)
356 * ruby_block[restart_log_service] action nothing (skipped due to action :nothing)
357 * ruby_block[reload_log_service] action nothing (skipped due to action :nothing)
358 * directory[/opt/gitlab/sv/logrotate] action create (up to date)
359 * template[/opt/gitlab/sv/logrotate/run] action create (up to date)
360 * directory[/opt/gitlab/sv/logrotate/log] action create (up to date)
361 * directory[/opt/gitlab/sv/logrotate/log/main] action create (up to date)
362 * template[/opt/gitlab/sv/logrotate/log/config] action create (up to date)
363 * ruby_block[verify_chown_persisted_on_logrotate] action nothing (skipped due to action :nothing)
364 * link[/var/log/gitlab/logrotate/config] action create (up to date)
365 * template[/opt/gitlab/sv/logrotate/log/run] action create (up to date)
366 * directory[/opt/gitlab/sv/logrotate/env] action create (up to date)
367 * ruby_block[Delete unmanaged env files for logrotate service] action run (skipped due to only_if)
368 * template[/opt/gitlab/sv/logrotate/check] action create (skipped due to only_if)
369 * template[/opt/gitlab/sv/logrotate/finish] action create (skipped due to only_if)
370 * directory[/opt/gitlab/sv/logrotate/control] action create (up to date)
371 * template[/opt/gitlab/sv/logrotate/control/t] action create (up to date)
372 * link[/opt/gitlab/init/logrotate] action create (up to date)
373 * file[/opt/gitlab/sv/logrotate/down] action nothing (skipped due to action :nothing)
374 * directory[/opt/gitlab/service] action create (up to date)
375 * link[/opt/gitlab/service/logrotate] action create[2023-07-20T03:20:23+00:00] INFO: link[/opt/gitlab/service/logrotate] created
376 - create symlink at /opt/gitlab/service/logrotate to /opt/gitlab/sv/logrotate
377 * ruby_block[wait for logrotate service socket] action run[2023-07-20T03:20:26+00:00] INFO: ruby_block[wait for logrotate service socket] called
378 - execute the ruby block wait for logrotate service socket
379 * file[/var/log/gitlab/logrotate/current] action touch (skipped due to only_if)
380[2023-07-20T03:20:26+00:00] INFO: ruby_block[reload_log_service] called
381 - execute the ruby block reload_log_service
382[2023-07-20T03:20:26+00:00] INFO: template[/opt/gitlab/sv/logrotate/log/run] sending run action to ruby_block[restart_log_service] (delayed)
383 * ruby_block[restart_log_service] action run
384 * ruby_block[restart_service] action nothing (skipped due to action :nothing)
385 * ruby_block[restart_log_service] action nothing (skipped due to action :nothing)
386 * ruby_block[reload_log_service] action nothing (skipped due to action :nothing)
387 * directory[/opt/gitlab/sv/logrotate] action create (up to date)
388 * template[/opt/gitlab/sv/logrotate/run] action create (up to date)
389 * directory[/opt/gitlab/sv/logrotate/log] action create (up to date)
390 * directory[/opt/gitlab/sv/logrotate/log/main] action create (up to date)
391 * template[/opt/gitlab/sv/logrotate/log/config] action create (up to date)
392 * ruby_block[verify_chown_persisted_on_logrotate] action nothing (skipped due to action :nothing)
393 * link[/var/log/gitlab/logrotate/config] action create (up to date)
394 * template[/opt/gitlab/sv/logrotate/log/run] action create (up to date)
395 * directory[/opt/gitlab/sv/logrotate/env] action create (up to date)
396 * ruby_block[Delete unmanaged env files for logrotate service] action run (skipped due to only_if)
397 * template[/opt/gitlab/sv/logrotate/check] action create (skipped due to only_if)
398 * template[/opt/gitlab/sv/logrotate/finish] action create (skipped due to only_if)
399 * directory[/opt/gitlab/sv/logrotate/control] action create (up to date)
400 * template[/opt/gitlab/sv/logrotate/control/t] action create (up to date)
401 * link[/opt/gitlab/init/logrotate] action create (up to date)
402 * file[/opt/gitlab/sv/logrotate/down] action nothing (skipped due to action :nothing)
403 * directory[/opt/gitlab/service] action create (up to date)
404 * link[/opt/gitlab/service/logrotate] action create (up to date)
405 * ruby_block[wait for logrotate service socket] action run (skipped due to not_if)
406 * file[/var/log/gitlab/logrotate/current] action touch (skipped due to only_if)
407[2023-07-20T03:20:27+00:00] INFO: ruby_block[restart_log_service] called
408 - execute the ruby block restart_log_service
409 * directory[/opt/gitlab/service] action create (up to date)
410 * link[/opt/gitlab/service/logrotate] action create (up to date)
411 * ruby_block[wait for logrotate service socket] action run (skipped due to not_if)
412 * file[/var/log/gitlab/logrotate/current] action touch (skipped due to only_if)
413
414 * execute[/opt/gitlab/bin/gitlab-ctl start logrotate] action run
415 [execute] ok: run: logrotate: (pid 534) 2s
416[2023-07-20T03:20:28+00:00] INFO: execute[/opt/gitlab/bin/gitlab-ctl start logrotate] ran successfully
417 - execute /opt/gitlab/bin/gitlab-ctl start logrotate
418Recipe: redis::disable
419 * service[redis] action nothing (skipped due to action :nothing)
420 * runit_service[redis] action disable
421 * ruby_block[disable redis] action run (skipped due to only_if)
422 (up to date)
423Recipe: gitaly::enable
424 * directory[/var/opt/gitlab/gitaly] action create[2023-07-20T03:20:28+00:00] INFO: directory[/var/opt/gitlab/gitaly] created directory /var/opt/gitlab/gitaly
425 - create new directory /var/opt/gitlab/gitaly[2023-07-20T03:20:28+00:00] INFO: directory[/var/opt/gitlab/gitaly] owner changed to 998
426[2023-07-20T03:20:28+00:00] INFO: directory[/var/opt/gitlab/gitaly] mode changed to 700
427 - change mode from '' to '0700'
428 - change owner from '' to 'git'
429 * directory[/var/opt/gitlab/gitaly/run] action create[2023-07-20T03:20:28+00:00] INFO: directory[/var/opt/gitlab/gitaly/run] created directory /var/opt/gitlab/gitaly/run
430 - create new directory /var/opt/gitlab/gitaly/run[2023-07-20T03:20:28+00:00] INFO: directory[/var/opt/gitlab/gitaly/run] owner changed to 998
431[2023-07-20T03:20:28+00:00] INFO: directory[/var/opt/gitlab/gitaly/run] mode changed to 700
432 - change mode from '' to '0700'
433 - change owner from '' to 'git'
434 * directory[/var/log/gitlab/gitaly] action create[2023-07-20T03:20:28+00:00] INFO: directory[/var/log/gitlab/gitaly] created directory /var/log/gitlab/gitaly
435 - create new directory /var/log/gitlab/gitaly[2023-07-20T03:20:28+00:00] INFO: directory[/var/log/gitlab/gitaly] owner changed to 998
436[2023-07-20T03:20:28+00:00] INFO: directory[/var/log/gitlab/gitaly] mode changed to 700
437 - change mode from '' to '0700'
438 - change owner from '' to 'git'
439 * directory[/var/opt/gitlab/gitaly/internal_sockets] action delete (up to date)
440 * env_dir[/opt/gitlab/etc/gitaly/env] action create
441 * directory[/opt/gitlab/etc/gitaly/env] action create[2023-07-20T03:20:28+00:00] INFO: directory[/opt/gitlab/etc/gitaly/env] created directory /opt/gitlab/etc/gitaly/env
442 - create new directory /opt/gitlab/etc/gitaly/env
443 * file[/opt/gitlab/etc/gitaly/env/HOME] action create[2023-07-20T03:20:28+00:00] INFO: file[/opt/gitlab/etc/gitaly/env/HOME] created file /opt/gitlab/etc/gitaly/env/HOME
444 - create new file /opt/gitlab/etc/gitaly/env/HOME[2023-07-20T03:20:28+00:00] INFO: file[/opt/gitlab/etc/gitaly/env/HOME] updated file contents /opt/gitlab/etc/gitaly/env/HOME
445 - update content in file /opt/gitlab/etc/gitaly/env/HOME from none to 205bb9
446 - suppressed sensitive resource
447 * file[/opt/gitlab/etc/gitaly/env/PATH] action create[2023-07-20T03:20:28+00:00] INFO: file[/opt/gitlab/etc/gitaly/env/PATH] created file /opt/gitlab/etc/gitaly/env/PATH
448 - create new file /opt/gitlab/etc/gitaly/env/PATH[2023-07-20T03:20:28+00:00] INFO: file[/opt/gitlab/etc/gitaly/env/PATH] updated file contents /opt/gitlab/etc/gitaly/env/PATH
449 - update content in file /opt/gitlab/etc/gitaly/env/PATH from none to d5dc07
450 - suppressed sensitive resource
451 * file[/opt/gitlab/etc/gitaly/env/TZ] action create[2023-07-20T03:20:28+00:00] INFO: file[/opt/gitlab/etc/gitaly/env/TZ] created file /opt/gitlab/etc/gitaly/env/TZ
452 - create new file /opt/gitlab/etc/gitaly/env/TZ[2023-07-20T03:20:28+00:00] INFO: file[/opt/gitlab/etc/gitaly/env/TZ] updated file contents /opt/gitlab/etc/gitaly/env/TZ
453 - update content in file /opt/gitlab/etc/gitaly/env/TZ from none to 983a95
454 - suppressed sensitive resource
455 * file[/opt/gitlab/etc/gitaly/env/PYTHONPATH] action create[2023-07-20T03:20:28+00:00] INFO: file[/opt/gitlab/etc/gitaly/env/PYTHONPATH] created file /opt/gitlab/etc/gitaly/env/PYTHONPATH
456 - create new file /opt/gitlab/etc/gitaly/env/PYTHONPATH[2023-07-20T03:20:28+00:00] INFO: file[/opt/gitlab/etc/gitaly/env/PYTHONPATH] updated file contents /opt/gitlab/etc/gitaly/env/PYTHONPATH
457 - update content in file /opt/gitlab/etc/gitaly/env/PYTHONPATH from none to 98c9f4
458 - suppressed sensitive resource
459 * file[/opt/gitlab/etc/gitaly/env/ICU_DATA] action create[2023-07-20T03:20:28+00:00] INFO: file[/opt/gitlab/etc/gitaly/env/ICU_DATA] created file /opt/gitlab/etc/gitaly/env/ICU_DATA
460 - create new file /opt/gitlab/etc/gitaly/env/ICU_DATA[2023-07-20T03:20:28+00:00] INFO: file[/opt/gitlab/etc/gitaly/env/ICU_DATA] updated file contents /opt/gitlab/etc/gitaly/env/ICU_DATA
461 - update content in file /opt/gitlab/etc/gitaly/env/ICU_DATA from none to a04260
462 - suppressed sensitive resource
463 * file[/opt/gitlab/etc/gitaly/env/SSL_CERT_DIR] action create[2023-07-20T03:20:28+00:00] INFO: file[/opt/gitlab/etc/gitaly/env/SSL_CERT_DIR] created file /opt/gitlab/etc/gitaly/env/SSL_CERT_DIR
464 - create new file /opt/gitlab/etc/gitaly/env/SSL_CERT_DIR[2023-07-20T03:20:28+00:00] INFO: file[/opt/gitlab/etc/gitaly/env/SSL_CERT_DIR] updated file contents /opt/gitlab/etc/gitaly/env/SSL_CERT_DIR
465 - update content in file /opt/gitlab/etc/gitaly/env/SSL_CERT_DIR from none to 4f45cf
466 - suppressed sensitive resource
467 * file[/opt/gitlab/etc/gitaly/env/GITALY_PID_FILE] action create[2023-07-20T03:20:28+00:00] INFO: file[/opt/gitlab/etc/gitaly/env/GITALY_PID_FILE] created file /opt/gitlab/etc/gitaly/env/GITALY_PID_FILE
468 - create new file /opt/gitlab/etc/gitaly/env/GITALY_PID_FILE[2023-07-20T03:20:28+00:00] INFO: file[/opt/gitlab/etc/gitaly/env/GITALY_PID_FILE] updated file contents /opt/gitlab/etc/gitaly/env/GITALY_PID_FILE
469 - update content in file /opt/gitlab/etc/gitaly/env/GITALY_PID_FILE from none to e82748
470 - suppressed sensitive resource
471 * file[/opt/gitlab/etc/gitaly/env/WRAPPER_JSON_LOGGING] action create[2023-07-20T03:20:28+00:00] INFO: file[/opt/gitlab/etc/gitaly/env/WRAPPER_JSON_LOGGING] created file /opt/gitlab/etc/gitaly/env/WRAPPER_JSON_LOGGING
472 - create new file /opt/gitlab/etc/gitaly/env/WRAPPER_JSON_LOGGING[2023-07-20T03:20:28+00:00] INFO: file[/opt/gitlab/etc/gitaly/env/WRAPPER_JSON_LOGGING] updated file contents /opt/gitlab/etc/gitaly/env/WRAPPER_JSON_LOGGING
473 - update content in file /opt/gitlab/etc/gitaly/env/WRAPPER_JSON_LOGGING from none to b5bea4
474 - suppressed sensitive resource
475
476 * template[Create Gitaly config.toml] action create[2023-07-20T03:20:28+00:00] INFO: template[Create Gitaly config.toml] created file /var/opt/gitlab/gitaly/config.toml
477 - create new file /var/opt/gitlab/gitaly/config.toml[2023-07-20T03:20:28+00:00] INFO: template[Create Gitaly config.toml] updated file contents /var/opt/gitlab/gitaly/config.toml
478 - update content in file /var/opt/gitlab/gitaly/config.toml from none to 6e396a
479 - suppressed sensitive resource[2023-07-20T03:20:28+00:00] INFO: template[Create Gitaly config.toml] owner changed to 0
480[2023-07-20T03:20:28+00:00] INFO: template[Create Gitaly config.toml] group changed to 998
481[2023-07-20T03:20:28+00:00] INFO: template[Create Gitaly config.toml] mode changed to 640
482 - change mode from '' to '0640'
483 - change owner from '' to 'root'
484 - change group from '' to 'git'
485 * service[gitaly] action nothing (skipped due to action :nothing)
486 * runit_service[gitaly] action enable
487 * ruby_block[restart_service] action nothing (skipped due to action :nothing)
488 * ruby_block[restart_log_service] action nothing (skipped due to action :nothing)
489 * ruby_block[reload_log_service] action nothing (skipped due to action :nothing)
490 * directory[/opt/gitlab/sv/gitaly] action create[2023-07-20T03:20:28+00:00] INFO: directory[/opt/gitlab/sv/gitaly] created directory /opt/gitlab/sv/gitaly
491 - create new directory /opt/gitlab/sv/gitaly[2023-07-20T03:20:28+00:00] INFO: directory[/opt/gitlab/sv/gitaly] owner changed to 0
492[2023-07-20T03:20:28+00:00] INFO: directory[/opt/gitlab/sv/gitaly] group changed to 0
493[2023-07-20T03:20:28+00:00] INFO: directory[/opt/gitlab/sv/gitaly] mode changed to 755
494 - change mode from '' to '0755'
495 - change owner from '' to 'root'
496 - change group from '' to 'root'
497 * template[/opt/gitlab/sv/gitaly/run] action create[2023-07-20T03:20:28+00:00] INFO: template[/opt/gitlab/sv/gitaly/run] created file /opt/gitlab/sv/gitaly/run
498 - create new file /opt/gitlab/sv/gitaly/run[2023-07-20T03:20:28+00:00] INFO: template[/opt/gitlab/sv/gitaly/run] updated file contents /opt/gitlab/sv/gitaly/run
499 - update content in file /opt/gitlab/sv/gitaly/run from none to 7fef11
500 --- /opt/gitlab/sv/gitaly/run 2023-07-20 03:20:28.000000000 +0000
501 +++ /opt/gitlab/sv/gitaly/.chef-run20230720-37-ify5tx 2023-07-20 03:20:28.000000000 +0000
502 @@ -1,19 +1,38 @@
503 +#!/bin/sh
504 +
505 +# Attempt to change ulimit before the set -e flag, ignore failures
506 +ulimit -n 15000
507 +
508 +set -e # fail on errors
509 +
510 +# Redirect stderr -> stdout
511 +exec 2>&1
512 +
513 +
514 +
515 +
516 +cd /var/opt/gitlab/gitaly
517 +
518 +exec chpst -e /opt/gitlab/etc/gitaly/env -P \
519 + -U git:git \
520 + -u git:git \
521 + /opt/gitlab/embedded/bin/gitaly-wrapper /opt/gitlab/embedded/bin/gitaly /var/opt/gitlab/gitaly/config.toml[2023-07-20T03:20:28+00:00] INFO: template[/opt/gitlab/sv/gitaly/run] owner changed to 0
522[2023-07-20T03:20:28+00:00] INFO: template[/opt/gitlab/sv/gitaly/run] group changed to 0
523[2023-07-20T03:20:28+00:00] INFO: template[/opt/gitlab/sv/gitaly/run] mode changed to 755
524 - change mode from '' to '0755'
525 - change owner from '' to 'root'
526 - change group from '' to 'root'
527 * directory[/opt/gitlab/sv/gitaly/log] action create[2023-07-20T03:20:28+00:00] INFO: directory[/opt/gitlab/sv/gitaly/log] created directory /opt/gitlab/sv/gitaly/log
528 - create new directory /opt/gitlab/sv/gitaly/log[2023-07-20T03:20:28+00:00] INFO: directory[/opt/gitlab/sv/gitaly/log] owner changed to 0
529[2023-07-20T03:20:28+00:00] INFO: directory[/opt/gitlab/sv/gitaly/log] group changed to 0
530[2023-07-20T03:20:28+00:00] INFO: directory[/opt/gitlab/sv/gitaly/log] mode changed to 755
531 - change mode from '' to '0755'
532 - change owner from '' to 'root'
533 - change group from '' to 'root'
534 * directory[/opt/gitlab/sv/gitaly/log/main] action create[2023-07-20T03:20:28+00:00] INFO: directory[/opt/gitlab/sv/gitaly/log/main] created directory /opt/gitlab/sv/gitaly/log/main
535 - create new directory /opt/gitlab/sv/gitaly/log/main[2023-07-20T03:20:28+00:00] INFO: directory[/opt/gitlab/sv/gitaly/log/main] owner changed to 0
536[2023-07-20T03:20:28+00:00] INFO: directory[/opt/gitlab/sv/gitaly/log/main] group changed to 0
537[2023-07-20T03:20:28+00:00] INFO: directory[/opt/gitlab/sv/gitaly/log/main] mode changed to 755
538 - change mode from '' to '0755'
539 - change owner from '' to 'root'
540 - change group from '' to 'root'
541 * template[/opt/gitlab/sv/gitaly/log/config] action create[2023-07-20T03:20:28+00:00] INFO: template[/opt/gitlab/sv/gitaly/log/config] created file /opt/gitlab/sv/gitaly/log/config
542 - create new file /opt/gitlab/sv/gitaly/log/config[2023-07-20T03:20:28+00:00] INFO: template[/opt/gitlab/sv/gitaly/log/config] updated file contents /opt/gitlab/sv/gitaly/log/config
543 - update content in file /opt/gitlab/sv/gitaly/log/config from none to 623c00
544 --- /opt/gitlab/sv/gitaly/log/config 2023-07-20 03:20:28.000000000 +0000
545 +++ /opt/gitlab/sv/gitaly/log/.chef-config20230720-37-fxmz7y 2023-07-20 03:20:28.000000000 +0000
546 @@ -1,6 +1,12 @@
547 +s209715200
548 +n30
549 +t86400
550 +!gzip
551 +
552 +[2023-07-20T03:20:28+00:00] INFO: template[/opt/gitlab/sv/gitaly/log/config] owner changed to 0
553[2023-07-20T03:20:28+00:00] INFO: template[/opt/gitlab/sv/gitaly/log/config] group changed to 0
554[2023-07-20T03:20:28+00:00] INFO: template[/opt/gitlab/sv/gitaly/log/config] mode changed to 644
555 - change mode from '' to '0644'
556 - change owner from '' to 'root'
557 - change group from '' to 'root'
558[2023-07-20T03:20:28+00:00] INFO: template[/opt/gitlab/sv/gitaly/log/config] sending create action to ruby_block[verify_chown_persisted_on_gitaly] (immediate)
559 * ruby_block[verify_chown_persisted_on_gitaly] action create[2023-07-20T03:20:28+00:00] INFO: ruby_block[verify_chown_persisted_on_gitaly] called
560 - execute the ruby block verify_chown_persisted_on_gitaly
561 * ruby_block[verify_chown_persisted_on_gitaly] action nothing (skipped due to action :nothing)
562 * link[/var/log/gitlab/gitaly/config] action create[2023-07-20T03:20:28+00:00] INFO: link[/var/log/gitlab/gitaly/config] created
563 - create symlink at /var/log/gitlab/gitaly/config to /opt/gitlab/sv/gitaly/log/config
564 * template[/opt/gitlab/sv/gitaly/log/run] action create[2023-07-20T03:20:28+00:00] INFO: template[/opt/gitlab/sv/gitaly/log/run] created file /opt/gitlab/sv/gitaly/log/run
565 - create new file /opt/gitlab/sv/gitaly/log/run[2023-07-20T03:20:28+00:00] INFO: template[/opt/gitlab/sv/gitaly/log/run] updated file contents /opt/gitlab/sv/gitaly/log/run
566 - update content in file /opt/gitlab/sv/gitaly/log/run from none to 209571
567 --- /opt/gitlab/sv/gitaly/log/run 2023-07-20 03:20:28.000000000 +0000
568 +++ /opt/gitlab/sv/gitaly/log/.chef-run20230720-37-2wrqlk 2023-07-20 03:20:28.000000000 +0000
569 @@ -1,5 +1,10 @@
570 +#!/bin/sh
571 +exec chpst -P \
572 + -U root:root \
573 + -u root:root \
574 + svlogd /var/log/gitlab/gitaly[2023-07-20T03:20:28+00:00] INFO: template[/opt/gitlab/sv/gitaly/log/run] owner changed to 0
575[2023-07-20T03:20:28+00:00] INFO: template[/opt/gitlab/sv/gitaly/log/run] group changed to 0
576[2023-07-20T03:20:28+00:00] INFO: template[/opt/gitlab/sv/gitaly/log/run] mode changed to 755
577 - change mode from '' to '0755'
578 - change owner from '' to 'root'
579 - change group from '' to 'root'
580 * directory[/opt/gitlab/sv/gitaly/env] action create[2023-07-20T03:20:28+00:00] INFO: directory[/opt/gitlab/sv/gitaly/env] created directory /opt/gitlab/sv/gitaly/env
581 - create new directory /opt/gitlab/sv/gitaly/env[2023-07-20T03:20:28+00:00] INFO: directory[/opt/gitlab/sv/gitaly/env] owner changed to 0
582[2023-07-20T03:20:28+00:00] INFO: directory[/opt/gitlab/sv/gitaly/env] group changed to 0
583[2023-07-20T03:20:28+00:00] INFO: directory[/opt/gitlab/sv/gitaly/env] mode changed to 755
584 - change mode from '' to '0755'
585 - change owner from '' to 'root'
586 - change group from '' to 'root'
587 * ruby_block[Delete unmanaged env files for gitaly service] action run (skipped due to only_if)
588 * template[/opt/gitlab/sv/gitaly/check] action create (skipped due to only_if)
589 * template[/opt/gitlab/sv/gitaly/finish] action create (skipped due to only_if)
590 * directory[/opt/gitlab/sv/gitaly/control] action create[2023-07-20T03:20:28+00:00] INFO: directory[/opt/gitlab/sv/gitaly/control] created directory /opt/gitlab/sv/gitaly/control
591 - create new directory /opt/gitlab/sv/gitaly/control[2023-07-20T03:20:28+00:00] INFO: directory[/opt/gitlab/sv/gitaly/control] owner changed to 0
592[2023-07-20T03:20:28+00:00] INFO: directory[/opt/gitlab/sv/gitaly/control] group changed to 0
593[2023-07-20T03:20:28+00:00] INFO: directory[/opt/gitlab/sv/gitaly/control] mode changed to 755
594 - change mode from '' to '0755'
595 - change owner from '' to 'root'
596 - change group from '' to 'root'
597 * link[/opt/gitlab/init/gitaly] action create[2023-07-20T03:20:28+00:00] INFO: link[/opt/gitlab/init/gitaly] created
598 - create symlink at /opt/gitlab/init/gitaly to /opt/gitlab/embedded/bin/sv
599 * file[/opt/gitlab/sv/gitaly/down] action nothing (skipped due to action :nothing)
600[2023-07-20T03:20:28+00:00] INFO: template[/opt/gitlab/sv/gitaly/run] sending run action to ruby_block[restart_service] (delayed)
601 * ruby_block[restart_service] action run (skipped due to only_if)
602[2023-07-20T03:20:28+00:00] INFO: template[/opt/gitlab/sv/gitaly/log/config] sending create action to ruby_block[reload_log_service] (delayed)
603 * ruby_block[reload_log_service] action create
604 * ruby_block[restart_service] action nothing (skipped due to action :nothing)
605 * ruby_block[restart_log_service] action nothing (skipped due to action :nothing)
606 * ruby_block[reload_log_service] action nothing (skipped due to action :nothing)
607 * directory[/opt/gitlab/sv/gitaly] action create (up to date)
608 * template[/opt/gitlab/sv/gitaly/run] action create (up to date)
609 * directory[/opt/gitlab/sv/gitaly/log] action create (up to date)
610 * directory[/opt/gitlab/sv/gitaly/log/main] action create (up to date)
611 * template[/opt/gitlab/sv/gitaly/log/config] action create (up to date)
612 * ruby_block[verify_chown_persisted_on_gitaly] action nothing (skipped due to action :nothing)
613 * link[/var/log/gitlab/gitaly/config] action create (up to date)
614 * template[/opt/gitlab/sv/gitaly/log/run] action create (up to date)
615 * directory[/opt/gitlab/sv/gitaly/env] action create (up to date)
616 * ruby_block[Delete unmanaged env files for gitaly service] action run (skipped due to only_if)
617 * template[/opt/gitlab/sv/gitaly/check] action create (skipped due to only_if)
618 * template[/opt/gitlab/sv/gitaly/finish] action create (skipped due to only_if)
619 * directory[/opt/gitlab/sv/gitaly/control] action create (up to date)
620 * link[/opt/gitlab/init/gitaly] action create (up to date)
621 * file[/opt/gitlab/sv/gitaly/down] action nothing (skipped due to action :nothing)
622 * directory[/opt/gitlab/service] action create (up to date)
623 * link[/opt/gitlab/service/gitaly] action create[2023-07-20T03:20:28+00:00] INFO: link[/opt/gitlab/service/gitaly] created
624 - create symlink at /opt/gitlab/service/gitaly to /opt/gitlab/sv/gitaly
625 * ruby_block[wait for gitaly service socket] action run[2023-07-20T03:20:32+00:00] INFO: ruby_block[wait for gitaly service socket] called
626 - execute the ruby block wait for gitaly service socket
627 * file[/var/log/gitlab/gitaly/current] action touch (skipped due to only_if)
628[2023-07-20T03:20:32+00:00] INFO: ruby_block[reload_log_service] called
629 - execute the ruby block reload_log_service
630[2023-07-20T03:20:32+00:00] INFO: template[/opt/gitlab/sv/gitaly/log/run] sending run action to ruby_block[restart_log_service] (delayed)
631 * ruby_block[restart_log_service] action run
632 * ruby_block[restart_service] action nothing (skipped due to action :nothing)
633 * ruby_block[restart_log_service] action nothing (skipped due to action :nothing)
634 * ruby_block[reload_log_service] action nothing (skipped due to action :nothing)
635 * directory[/opt/gitlab/sv/gitaly] action create (up to date)
636 * template[/opt/gitlab/sv/gitaly/run] action create (up to date)
637 * directory[/opt/gitlab/sv/gitaly/log] action create (up to date)
638 * directory[/opt/gitlab/sv/gitaly/log/main] action create (up to date)
639 * template[/opt/gitlab/sv/gitaly/log/config] action create (up to date)
640 * ruby_block[verify_chown_persisted_on_gitaly] action nothing (skipped due to action :nothing)
641 * link[/var/log/gitlab/gitaly/config] action create (up to date)
642 * template[/opt/gitlab/sv/gitaly/log/run] action create (up to date)
643 * directory[/opt/gitlab/sv/gitaly/env] action create (up to date)
644 * ruby_block[Delete unmanaged env files for gitaly service] action run (skipped due to only_if)
645 * template[/opt/gitlab/sv/gitaly/check] action create (skipped due to only_if)
646 * template[/opt/gitlab/sv/gitaly/finish] action create (skipped due to only_if)
647 * directory[/opt/gitlab/sv/gitaly/control] action create (up to date)
648 * link[/opt/gitlab/init/gitaly] action create (up to date)
649 * file[/opt/gitlab/sv/gitaly/down] action nothing (skipped due to action :nothing)
650 * directory[/opt/gitlab/service] action create (up to date)
651 * link[/opt/gitlab/service/gitaly] action create (up to date)
652 * ruby_block[wait for gitaly service socket] action run (skipped due to not_if)
653 * file[/var/log/gitlab/gitaly/current] action touch (skipped due to only_if)
654[2023-07-20T03:20:34+00:00] INFO: ruby_block[restart_log_service] called
655 - execute the ruby block restart_log_service
656 * directory[/opt/gitlab/service] action create (up to date)
657 * link[/opt/gitlab/service/gitaly] action create (up to date)
658 * ruby_block[wait for gitaly service socket] action run (skipped due to not_if)
659 * file[/var/log/gitlab/gitaly/current] action touch (skipped due to only_if)
660
661 * execute[/opt/gitlab/bin/gitlab-ctl start gitaly] action run
662 [execute] ok: run: gitaly: (pid 550) 2s
663[2023-07-20T03:20:34+00:00] INFO: execute[/opt/gitlab/bin/gitlab-ctl start gitaly] ran successfully
664 - execute /opt/gitlab/bin/gitlab-ctl start gitaly
665 * version_file[Create version file for Gitaly] action create
666 * file[/var/opt/gitlab/gitaly/VERSION] action create[2023-07-20T03:20:36+00:00] INFO: file[/var/opt/gitlab/gitaly/VERSION] created file /var/opt/gitlab/gitaly/VERSION
667 - create new file /var/opt/gitlab/gitaly/VERSION[2023-07-20T03:20:36+00:00] INFO: file[/var/opt/gitlab/gitaly/VERSION] updated file contents /var/opt/gitlab/gitaly/VERSION
668 - update content in file /var/opt/gitlab/gitaly/VERSION from none to 7775d5
669 --- /var/opt/gitlab/gitaly/VERSION 2023-07-20 03:20:36.000000000 +0000
670 +++ /var/opt/gitlab/gitaly/.chef-VERSION20230720-37-bol8tt 2023-07-20 03:20:36.000000000 +0000
671 @@ -1 +1,2 @@
672 +sha256:1523691d349404809e5fcf02cc8b6e4429851cf83954a7e4be29a36b3b9e225e
673
674 * consul_service[gitaly] action delete
675 * file[/var/opt/gitlab/consul/config.d/gitaly-service.json] action delete (up to date)
676 (up to date)
677Recipe: postgresql::bin
678 * ruby_block[check_postgresql_version] action run (skipped due to not_if)
679 * ruby_block[check_postgresql_version_is_deprecated] action run (skipped due to not_if)
680 * ruby_block[Link postgresql bin files to the correct version] action run[2023-07-20T03:20:36+00:00] INFO: ruby_block[Link postgresql bin files to the correct version] called
681 - execute the ruby block Link postgresql bin files to the correct version
682 * template[/opt/gitlab/etc/gitlab-psql-rc] action create[2023-07-20T03:20:36+00:00] INFO: template[/opt/gitlab/etc/gitlab-psql-rc] created file /opt/gitlab/etc/gitlab-psql-rc
683 - create new file /opt/gitlab/etc/gitlab-psql-rc[2023-07-20T03:20:36+00:00] INFO: template[/opt/gitlab/etc/gitlab-psql-rc] updated file contents /opt/gitlab/etc/gitlab-psql-rc
684 - update content in file /opt/gitlab/etc/gitlab-psql-rc from none to b7b8fc
685 - suppressed sensitive resource[2023-07-20T03:20:36+00:00] INFO: template[/opt/gitlab/etc/gitlab-psql-rc] owner changed to 0
686[2023-07-20T03:20:36+00:00] INFO: template[/opt/gitlab/etc/gitlab-psql-rc] group changed to 0
687 - change owner from '' to 'root'
688 - change group from '' to 'root'
689Recipe: postgresql::disable
690 * service[postgresql] action nothing (skipped due to action :nothing)
691 * runit_service[postgresql] action disable
692 * ruby_block[disable postgresql] action run (skipped due to only_if)
693 (up to date)
694Recipe: praefect::disable
695 * service[praefect] action nothing (skipped due to action :nothing)
696 * runit_service[praefect] action disable
697 * ruby_block[disable praefect] action run (skipped due to only_if)
698 (up to date)
699 * consul_service[praefect] action delete
700 * file[/var/opt/gitlab/consul/config.d/praefect-service.json] action delete (up to date)
701 (up to date)
702Recipe: gitlab-kas::enable
703 * directory[/var/opt/gitlab/gitlab-kas] action create[2023-07-20T03:20:36+00:00] INFO: directory[/var/opt/gitlab/gitlab-kas] created directory /var/opt/gitlab/gitlab-kas
704 - create new directory /var/opt/gitlab/gitlab-kas[2023-07-20T03:20:36+00:00] INFO: directory[/var/opt/gitlab/gitlab-kas] owner changed to 998
705[2023-07-20T03:20:36+00:00] INFO: directory[/var/opt/gitlab/gitlab-kas] mode changed to 700
706 - change mode from '' to '0700'
707 - change owner from '' to 'git'
708 * directory[/opt/gitlab/etc/gitlab-kas] action create[2023-07-20T03:20:36+00:00] INFO: directory[/opt/gitlab/etc/gitlab-kas] created directory /opt/gitlab/etc/gitlab-kas
709 - create new directory /opt/gitlab/etc/gitlab-kas[2023-07-20T03:20:36+00:00] INFO: directory[/opt/gitlab/etc/gitlab-kas] owner changed to 998
710[2023-07-20T03:20:36+00:00] INFO: directory[/opt/gitlab/etc/gitlab-kas] mode changed to 700
711 - change mode from '' to '0700'
712 - change owner from '' to 'git'
713 * directory[/var/log/gitlab/gitlab-kas] action create[2023-07-20T03:20:36+00:00] INFO: directory[/var/log/gitlab/gitlab-kas] created directory /var/log/gitlab/gitlab-kas
714 - create new directory /var/log/gitlab/gitlab-kas[2023-07-20T03:20:36+00:00] INFO: directory[/var/log/gitlab/gitlab-kas] owner changed to 998
715[2023-07-20T03:20:36+00:00] INFO: directory[/var/log/gitlab/gitlab-kas] mode changed to 700
716 - change mode from '' to '0700'
717 - change owner from '' to 'git'
718 * ruby_block[websocket TLS termination] action run (skipped due to only_if)
719 * version_file[Create version file for Gitlab KAS] action create
720 * file[/var/opt/gitlab/gitlab-kas/VERSION] action create[2023-07-20T03:20:36+00:00] INFO: file[/var/opt/gitlab/gitlab-kas/VERSION] created file /var/opt/gitlab/gitlab-kas/VERSION
721 - create new file /var/opt/gitlab/gitlab-kas/VERSION[2023-07-20T03:20:36+00:00] INFO: file[/var/opt/gitlab/gitlab-kas/VERSION] updated file contents /var/opt/gitlab/gitlab-kas/VERSION
722 - update content in file /var/opt/gitlab/gitlab-kas/VERSION from none to d3f397
723 --- /var/opt/gitlab/gitlab-kas/VERSION 2023-07-20 03:20:36.000000000 +0000
724 +++ /var/opt/gitlab/gitlab-kas/.chef-VERSION20230720-37-9f4vuq 2023-07-20 03:20:36.000000000 +0000
725 @@ -1 +1,2 @@
726 +kas version v16.2.0, commit: 79f3b36c, built: 20230720.023210
727
728 * file[/var/opt/gitlab/gitlab-kas/authentication_secret_file] action create[2023-07-20T03:20:36+00:00] INFO: file[/var/opt/gitlab/gitlab-kas/authentication_secret_file] created file /var/opt/gitlab/gitlab-kas/authentication_secret_file
729 - create new file /var/opt/gitlab/gitlab-kas/authentication_secret_file[2023-07-20T03:20:36+00:00] INFO: file[/var/opt/gitlab/gitlab-kas/authentication_secret_file] updated file contents /var/opt/gitlab/gitlab-kas/authentication_secret_file
730 - update content in file /var/opt/gitlab/gitlab-kas/authentication_secret_file from none to 43d396
731 --- /var/opt/gitlab/gitlab-kas/authentication_secret_file 2023-07-20 03:20:36.000000000 +0000
732 +++ /var/opt/gitlab/gitlab-kas/.chef-authentication_secret_file20230720-37-ulg3hu 2023-07-20 03:20:36.000000000 +0000
733 @@ -1 +1,2 @@
734 +NjZmYmYwMjhhMzQ1ZDU1YjhhZDE0ZmYxNzA3ZmI2MzM=[2023-07-20T03:20:36+00:00] INFO: file[/var/opt/gitlab/gitlab-kas/authentication_secret_file] owner changed to 0
735[2023-07-20T03:20:36+00:00] INFO: file[/var/opt/gitlab/gitlab-kas/authentication_secret_file] group changed to 998
736[2023-07-20T03:20:36+00:00] INFO: file[/var/opt/gitlab/gitlab-kas/authentication_secret_file] mode changed to 640
737 - change mode from '' to '0640'
738 - change owner from '' to 'root'
739 - change group from '' to 'git'
740[2023-07-20T03:20:36+00:00] INFO: file[/var/opt/gitlab/gitlab-kas/authentication_secret_file] not queuing delayed action restart on runit_service[gitlab-kas] (delayed), as it's already been queued
741 * file[/var/opt/gitlab/gitlab-kas/private_api_authentication_secret_file] action create[2023-07-20T03:20:36+00:00] INFO: file[/var/opt/gitlab/gitlab-kas/private_api_authentication_secret_file] created file /var/opt/gitlab/gitlab-kas/private_api_authentication_secret_file
742 - create new file /var/opt/gitlab/gitlab-kas/private_api_authentication_secret_file[2023-07-20T03:20:36+00:00] INFO: file[/var/opt/gitlab/gitlab-kas/private_api_authentication_secret_file] updated file contents /var/opt/gitlab/gitlab-kas/private_api_authentication_secret_file
743 - update content in file /var/opt/gitlab/gitlab-kas/private_api_authentication_secret_file from none to e3c0bc
744 --- /var/opt/gitlab/gitlab-kas/private_api_authentication_secret_file 2023-07-20 03:20:36.000000000 +0000
745 +++ /var/opt/gitlab/gitlab-kas/.chef-private_api_authentication_secret_file20230720-37-g0nb3u 2023-07-20 03:20:36.000000000 +0000
746 @@ -1 +1,2 @@
747 +Nzk2OTJkNDhkMTVkZWZhZGU2MjVkODIzYzMyYjVlMDA=[2023-07-20T03:20:36+00:00] INFO: file[/var/opt/gitlab/gitlab-kas/private_api_authentication_secret_file] owner changed to 0
748[2023-07-20T03:20:36+00:00] INFO: file[/var/opt/gitlab/gitlab-kas/private_api_authentication_secret_file] group changed to 998
749[2023-07-20T03:20:36+00:00] INFO: file[/var/opt/gitlab/gitlab-kas/private_api_authentication_secret_file] mode changed to 640
750 - change mode from '' to '0640'
751 - change owner from '' to 'root'
752 - change group from '' to 'git'
753[2023-07-20T03:20:36+00:00] INFO: file[/var/opt/gitlab/gitlab-kas/private_api_authentication_secret_file] not queuing delayed action restart on runit_service[gitlab-kas] (delayed), as it's already been queued
754 * file[/var/opt/gitlab/gitlab-kas/redis_password_file] action create (skipped due to only_if)
755 * file[/var/opt/gitlab/gitlab-kas/redis_sentinels_password_file] action create (skipped due to only_if)
756 * template[/var/opt/gitlab/gitlab-kas/gitlab-kas-config.yml] action create[2023-07-20T03:20:36+00:00] INFO: template[/var/opt/gitlab/gitlab-kas/gitlab-kas-config.yml] created file /var/opt/gitlab/gitlab-kas/gitlab-kas-config.yml
757 - create new file /var/opt/gitlab/gitlab-kas/gitlab-kas-config.yml[2023-07-20T03:20:36+00:00] INFO: template[/var/opt/gitlab/gitlab-kas/gitlab-kas-config.yml] updated file contents /var/opt/gitlab/gitlab-kas/gitlab-kas-config.yml
758 - update content in file /var/opt/gitlab/gitlab-kas/gitlab-kas-config.yml from none to 3bf209
759 --- /var/opt/gitlab/gitlab-kas/gitlab-kas-config.yml 2023-07-20 03:20:36.000000000 +0000
760 +++ /var/opt/gitlab/gitlab-kas/.chef-gitlab-kas-config20230720-37-sokiwg.yml 2023-07-20 03:20:36.000000000 +0000
761 @@ -1,44 +1,88 @@
762 +agent:
763 + listen:
764 + address: localhost:8150
765 + network: tcp
766 + websocket: true
767 + configuration:
768 + poll_period: 20s
769 + gitops:
770 + poll_period: 20s
771 + project_info_cache_ttl: 300s
772 + project_info_cache_error_ttl: 60s
773 + kubernetes_api:
774 + listen:
775 + address: localhost:8154
776 + url_path_prefix: /
777 + info_cache_ttl: 300s
778 + info_cache_error_ttl: 60s
779 +gitlab:
780 + address: http://gitaly3.test
781 + external_url: http://gitaly3.test
782 + authentication_secret_file: /var/opt/gitlab/gitlab-kas/authentication_secret_file
783 +observability:
784 + listen:
785 + address: localhost:8151
786 + network: tcp
787 + usage_reporting_period: 60s
788 + logging:
789 + level: info
790 +redis:
791 + network: unix
792 + tls:
793 + enabled: false
794 + server:
795 + address: '/var/opt/gitlab/redis/redis.socket'
796 +api:
797 + listen:
798 + network: tcp
799 + address: localhost:8153
800 + authentication_secret_file: /var/opt/gitlab/gitlab-kas/authentication_secret_file
801 +private_api:
802 + listen:
803 + network: tcp
804 + address: localhost:8155
805 + authentication_secret_file: /var/opt/gitlab/gitlab-kas/private_api_authentication_secret_file[2023-07-20T03:20:36+00:00] INFO: template[/var/opt/gitlab/gitlab-kas/gitlab-kas-config.yml] owner changed to 0
806[2023-07-20T03:20:36+00:00] INFO: template[/var/opt/gitlab/gitlab-kas/gitlab-kas-config.yml] group changed to 998
807[2023-07-20T03:20:36+00:00] INFO: template[/var/opt/gitlab/gitlab-kas/gitlab-kas-config.yml] mode changed to 640
808 - change mode from '' to '0640'
809 - change owner from '' to 'root'
810 - change group from '' to 'git'
811[2023-07-20T03:20:36+00:00] INFO: template[/var/opt/gitlab/gitlab-kas/gitlab-kas-config.yml] not queuing delayed action restart on runit_service[gitlab-kas] (delayed), as it's already been queued
812 * env_dir[/opt/gitlab/etc/gitlab-kas/env] action create
813 * directory[/opt/gitlab/etc/gitlab-kas/env] action create[2023-07-20T03:20:36+00:00] INFO: directory[/opt/gitlab/etc/gitlab-kas/env] created directory /opt/gitlab/etc/gitlab-kas/env
814 - create new directory /opt/gitlab/etc/gitlab-kas/env
815 * file[/opt/gitlab/etc/gitlab-kas/env/SSL_CERT_DIR] action create[2023-07-20T03:20:36+00:00] INFO: file[/opt/gitlab/etc/gitlab-kas/env/SSL_CERT_DIR] created file /opt/gitlab/etc/gitlab-kas/env/SSL_CERT_DIR
816 - create new file /opt/gitlab/etc/gitlab-kas/env/SSL_CERT_DIR[2023-07-20T03:20:36+00:00] INFO: file[/opt/gitlab/etc/gitlab-kas/env/SSL_CERT_DIR] updated file contents /opt/gitlab/etc/gitlab-kas/env/SSL_CERT_DIR
817 - update content in file /opt/gitlab/etc/gitlab-kas/env/SSL_CERT_DIR from none to 4f45cf
818 - suppressed sensitive resource
819 * file[/opt/gitlab/etc/gitlab-kas/env/OWN_PRIVATE_API_URL] action create[2023-07-20T03:20:36+00:00] INFO: file[/opt/gitlab/etc/gitlab-kas/env/OWN_PRIVATE_API_URL] created file /opt/gitlab/etc/gitlab-kas/env/OWN_PRIVATE_API_URL
820 - create new file /opt/gitlab/etc/gitlab-kas/env/OWN_PRIVATE_API_URL[2023-07-20T03:20:36+00:00] INFO: file[/opt/gitlab/etc/gitlab-kas/env/OWN_PRIVATE_API_URL] updated file contents /opt/gitlab/etc/gitlab-kas/env/OWN_PRIVATE_API_URL
821 - update content in file /opt/gitlab/etc/gitlab-kas/env/OWN_PRIVATE_API_URL from none to 45b898
822 - suppressed sensitive resource
823
824 * service[gitlab-kas] action nothing (skipped due to action :nothing)
825 * runit_service[gitlab-kas] action enable
826 * ruby_block[restart_service] action nothing (skipped due to action :nothing)
827 * ruby_block[restart_log_service] action nothing (skipped due to action :nothing)
828 * ruby_block[reload_log_service] action nothing (skipped due to action :nothing)
829 * directory[/opt/gitlab/sv/gitlab-kas] action create[2023-07-20T03:20:36+00:00] INFO: directory[/opt/gitlab/sv/gitlab-kas] created directory /opt/gitlab/sv/gitlab-kas
830 - create new directory /opt/gitlab/sv/gitlab-kas[2023-07-20T03:20:36+00:00] INFO: directory[/opt/gitlab/sv/gitlab-kas] owner changed to 0
831[2023-07-20T03:20:36+00:00] INFO: directory[/opt/gitlab/sv/gitlab-kas] group changed to 0
832[2023-07-20T03:20:36+00:00] INFO: directory[/opt/gitlab/sv/gitlab-kas] mode changed to 755
833 - change mode from '' to '0755'
834 - change owner from '' to 'root'
835 - change group from '' to 'root'
836 * template[/opt/gitlab/sv/gitlab-kas/run] action create[2023-07-20T03:20:36+00:00] INFO: template[/opt/gitlab/sv/gitlab-kas/run] created file /opt/gitlab/sv/gitlab-kas/run
837 - create new file /opt/gitlab/sv/gitlab-kas/run[2023-07-20T03:20:36+00:00] INFO: template[/opt/gitlab/sv/gitlab-kas/run] updated file contents /opt/gitlab/sv/gitlab-kas/run
838 - update content in file /opt/gitlab/sv/gitlab-kas/run from none to ac037b
839 --- /opt/gitlab/sv/gitlab-kas/run 2023-07-20 03:20:36.000000000 +0000
840 +++ /opt/gitlab/sv/gitlab-kas/.chef-run20230720-37-ev90fc 2023-07-20 03:20:36.000000000 +0000
841 @@ -1,11 +1,22 @@
842 +#!/bin/bash
843 +
844 +# Let runit capture all script error messages
845 +exec 2>&1
846 +
847 +
848 +
849 +exec chpst -e /opt/gitlab/etc/gitlab-kas/env -P \
850 + -u git:git \
851 + -U git:git \
852 + /opt/gitlab/embedded/bin/gitlab-kas --configuration-file /var/opt/gitlab/gitlab-kas/gitlab-kas-config.yml[2023-07-20T03:20:36+00:00] INFO: template[/opt/gitlab/sv/gitlab-kas/run] owner changed to 0
853[2023-07-20T03:20:36+00:00] INFO: template[/opt/gitlab/sv/gitlab-kas/run] group changed to 0
854[2023-07-20T03:20:36+00:00] INFO: template[/opt/gitlab/sv/gitlab-kas/run] mode changed to 755
855 - change mode from '' to '0755'
856 - change owner from '' to 'root'
857 - change group from '' to 'root'
858 * directory[/opt/gitlab/sv/gitlab-kas/log] action create[2023-07-20T03:20:36+00:00] INFO: directory[/opt/gitlab/sv/gitlab-kas/log] created directory /opt/gitlab/sv/gitlab-kas/log
859 - create new directory /opt/gitlab/sv/gitlab-kas/log[2023-07-20T03:20:36+00:00] INFO: directory[/opt/gitlab/sv/gitlab-kas/log] owner changed to 0
860[2023-07-20T03:20:36+00:00] INFO: directory[/opt/gitlab/sv/gitlab-kas/log] group changed to 0
861[2023-07-20T03:20:36+00:00] INFO: directory[/opt/gitlab/sv/gitlab-kas/log] mode changed to 755
862 - change mode from '' to '0755'
863 - change owner from '' to 'root'
864 - change group from '' to 'root'
865 * directory[/opt/gitlab/sv/gitlab-kas/log/main] action create[2023-07-20T03:20:36+00:00] INFO: directory[/opt/gitlab/sv/gitlab-kas/log/main] created directory /opt/gitlab/sv/gitlab-kas/log/main
866 - create new directory /opt/gitlab/sv/gitlab-kas/log/main[2023-07-20T03:20:36+00:00] INFO: directory[/opt/gitlab/sv/gitlab-kas/log/main] owner changed to 0
867[2023-07-20T03:20:36+00:00] INFO: directory[/opt/gitlab/sv/gitlab-kas/log/main] group changed to 0
868[2023-07-20T03:20:36+00:00] INFO: directory[/opt/gitlab/sv/gitlab-kas/log/main] mode changed to 755
869 - change mode from '' to '0755'
870 - change owner from '' to 'root'
871 - change group from '' to 'root'
872 * template[/opt/gitlab/sv/gitlab-kas/log/config] action create[2023-07-20T03:20:36+00:00] INFO: template[/opt/gitlab/sv/gitlab-kas/log/config] created file /opt/gitlab/sv/gitlab-kas/log/config
873 - create new file /opt/gitlab/sv/gitlab-kas/log/config[2023-07-20T03:20:36+00:00] INFO: template[/opt/gitlab/sv/gitlab-kas/log/config] updated file contents /opt/gitlab/sv/gitlab-kas/log/config
874 - update content in file /opt/gitlab/sv/gitlab-kas/log/config from none to 623c00
875 --- /opt/gitlab/sv/gitlab-kas/log/config 2023-07-20 03:20:36.000000000 +0000
876 +++ /opt/gitlab/sv/gitlab-kas/log/.chef-config20230720-37-xmobim 2023-07-20 03:20:36.000000000 +0000
877 @@ -1,6 +1,12 @@
878 +s209715200
879 +n30
880 +t86400
881 +!gzip
882 +
883 +[2023-07-20T03:20:36+00:00] INFO: template[/opt/gitlab/sv/gitlab-kas/log/config] owner changed to 0
884[2023-07-20T03:20:36+00:00] INFO: template[/opt/gitlab/sv/gitlab-kas/log/config] group changed to 0
885[2023-07-20T03:20:36+00:00] INFO: template[/opt/gitlab/sv/gitlab-kas/log/config] mode changed to 644
886 - change mode from '' to '0644'
887 - change owner from '' to 'root'
888 - change group from '' to 'root'
889[2023-07-20T03:20:36+00:00] INFO: template[/opt/gitlab/sv/gitlab-kas/log/config] sending create action to ruby_block[verify_chown_persisted_on_gitlab-kas] (immediate)
890 * ruby_block[verify_chown_persisted_on_gitlab-kas] action create[2023-07-20T03:20:36+00:00] INFO: ruby_block[verify_chown_persisted_on_gitlab-kas] called
891 - execute the ruby block verify_chown_persisted_on_gitlab-kas
892 * ruby_block[verify_chown_persisted_on_gitlab-kas] action nothing (skipped due to action :nothing)
893 * link[/var/log/gitlab/gitlab-kas/config] action create[2023-07-20T03:20:36+00:00] INFO: link[/var/log/gitlab/gitlab-kas/config] created
894 - create symlink at /var/log/gitlab/gitlab-kas/config to /opt/gitlab/sv/gitlab-kas/log/config
895 * template[/opt/gitlab/sv/gitlab-kas/log/run] action create[2023-07-20T03:20:36+00:00] INFO: template[/opt/gitlab/sv/gitlab-kas/log/run] created file /opt/gitlab/sv/gitlab-kas/log/run
896 - create new file /opt/gitlab/sv/gitlab-kas/log/run[2023-07-20T03:20:36+00:00] INFO: template[/opt/gitlab/sv/gitlab-kas/log/run] updated file contents /opt/gitlab/sv/gitlab-kas/log/run
897 - update content in file /opt/gitlab/sv/gitlab-kas/log/run from none to 22436c
898 --- /opt/gitlab/sv/gitlab-kas/log/run 2023-07-20 03:20:36.000000000 +0000
899 +++ /opt/gitlab/sv/gitlab-kas/log/.chef-run20230720-37-i7asbp 2023-07-20 03:20:36.000000000 +0000
900 @@ -1,5 +1,10 @@
901 +#!/bin/sh
902 +exec chpst -P \
903 + -U root:root \
904 + -u root:root \
905 + svlogd -tt /var/log/gitlab/gitlab-kas[2023-07-20T03:20:36+00:00] INFO: template[/opt/gitlab/sv/gitlab-kas/log/run] owner changed to 0
906[2023-07-20T03:20:36+00:00] INFO: template[/opt/gitlab/sv/gitlab-kas/log/run] group changed to 0
907[2023-07-20T03:20:36+00:00] INFO: template[/opt/gitlab/sv/gitlab-kas/log/run] mode changed to 755
908 - change mode from '' to '0755'
909 - change owner from '' to 'root'
910 - change group from '' to 'root'
911 * directory[/opt/gitlab/sv/gitlab-kas/env] action create[2023-07-20T03:20:36+00:00] INFO: directory[/opt/gitlab/sv/gitlab-kas/env] created directory /opt/gitlab/sv/gitlab-kas/env
912 - create new directory /opt/gitlab/sv/gitlab-kas/env[2023-07-20T03:20:36+00:00] INFO: directory[/opt/gitlab/sv/gitlab-kas/env] owner changed to 0
913[2023-07-20T03:20:36+00:00] INFO: directory[/opt/gitlab/sv/gitlab-kas/env] group changed to 0
914[2023-07-20T03:20:36+00:00] INFO: directory[/opt/gitlab/sv/gitlab-kas/env] mode changed to 755
915 - change mode from '' to '0755'
916 - change owner from '' to 'root'
917 - change group from '' to 'root'
918 * ruby_block[Delete unmanaged env files for gitlab-kas service] action run (skipped due to only_if)
919 * template[/opt/gitlab/sv/gitlab-kas/check] action create (skipped due to only_if)
920 * template[/opt/gitlab/sv/gitlab-kas/finish] action create (skipped due to only_if)
921 * directory[/opt/gitlab/sv/gitlab-kas/control] action create[2023-07-20T03:20:36+00:00] INFO: directory[/opt/gitlab/sv/gitlab-kas/control] created directory /opt/gitlab/sv/gitlab-kas/control
922 - create new directory /opt/gitlab/sv/gitlab-kas/control[2023-07-20T03:20:36+00:00] INFO: directory[/opt/gitlab/sv/gitlab-kas/control] owner changed to 0
923[2023-07-20T03:20:36+00:00] INFO: directory[/opt/gitlab/sv/gitlab-kas/control] group changed to 0
924[2023-07-20T03:20:36+00:00] INFO: directory[/opt/gitlab/sv/gitlab-kas/control] mode changed to 755
925 - change mode from '' to '0755'
926 - change owner from '' to 'root'
927 - change group from '' to 'root'
928 * link[/opt/gitlab/init/gitlab-kas] action create[2023-07-20T03:20:36+00:00] INFO: link[/opt/gitlab/init/gitlab-kas] created
929 - create symlink at /opt/gitlab/init/gitlab-kas to /opt/gitlab/embedded/bin/sv
930 * file[/opt/gitlab/sv/gitlab-kas/down] action nothing (skipped due to action :nothing)
931[2023-07-20T03:20:36+00:00] INFO: template[/opt/gitlab/sv/gitlab-kas/run] sending run action to ruby_block[restart_service] (delayed)
932 * ruby_block[restart_service] action run (skipped due to only_if)
933[2023-07-20T03:20:36+00:00] INFO: template[/opt/gitlab/sv/gitlab-kas/log/config] sending create action to ruby_block[reload_log_service] (delayed)
934 * ruby_block[reload_log_service] action create
935 * ruby_block[restart_service] action nothing (skipped due to action :nothing)
936 * ruby_block[restart_log_service] action nothing (skipped due to action :nothing)
937 * ruby_block[reload_log_service] action nothing (skipped due to action :nothing)
938 * directory[/opt/gitlab/sv/gitlab-kas] action create (up to date)
939 * template[/opt/gitlab/sv/gitlab-kas/run] action create (up to date)
940 * directory[/opt/gitlab/sv/gitlab-kas/log] action create (up to date)
941 * directory[/opt/gitlab/sv/gitlab-kas/log/main] action create (up to date)
942 * template[/opt/gitlab/sv/gitlab-kas/log/config] action create (up to date)
943 * ruby_block[verify_chown_persisted_on_gitlab-kas] action nothing (skipped due to action :nothing)
944 * link[/var/log/gitlab/gitlab-kas/config] action create (up to date)
945 * template[/opt/gitlab/sv/gitlab-kas/log/run] action create (up to date)
946 * directory[/opt/gitlab/sv/gitlab-kas/env] action create (up to date)
947 * ruby_block[Delete unmanaged env files for gitlab-kas service] action run (skipped due to only_if)
948 * template[/opt/gitlab/sv/gitlab-kas/check] action create (skipped due to only_if)
949 * template[/opt/gitlab/sv/gitlab-kas/finish] action create (skipped due to only_if)
950 * directory[/opt/gitlab/sv/gitlab-kas/control] action create (up to date)
951 * link[/opt/gitlab/init/gitlab-kas] action create (up to date)
952 * file[/opt/gitlab/sv/gitlab-kas/down] action nothing (skipped due to action :nothing)
953 * directory[/opt/gitlab/service] action create (up to date)
954 * link[/opt/gitlab/service/gitlab-kas] action create[2023-07-20T03:20:36+00:00] INFO: link[/opt/gitlab/service/gitlab-kas] created
955 - create symlink at /opt/gitlab/service/gitlab-kas to /opt/gitlab/sv/gitlab-kas
956 * ruby_block[wait for gitlab-kas service socket] action run[2023-07-20T03:20:38+00:00] INFO: ruby_block[wait for gitlab-kas service socket] called
957 - execute the ruby block wait for gitlab-kas service socket
958 * file[/var/log/gitlab/gitlab-kas/current] action touch (skipped due to only_if)
959[2023-07-20T03:20:38+00:00] INFO: ruby_block[reload_log_service] called
960 - execute the ruby block reload_log_service
961[2023-07-20T03:20:38+00:00] INFO: template[/opt/gitlab/sv/gitlab-kas/log/run] sending run action to ruby_block[restart_log_service] (delayed)
962 * ruby_block[restart_log_service] action run
963 * ruby_block[restart_service] action nothing (skipped due to action :nothing)
964 * ruby_block[restart_log_service] action nothing (skipped due to action :nothing)
965 * ruby_block[reload_log_service] action nothing (skipped due to action :nothing)
966 * directory[/opt/gitlab/sv/gitlab-kas] action create (up to date)
967 * template[/opt/gitlab/sv/gitlab-kas/run] action create (up to date)
968 * directory[/opt/gitlab/sv/gitlab-kas/log] action create (up to date)
969 * directory[/opt/gitlab/sv/gitlab-kas/log/main] action create (up to date)
970 * template[/opt/gitlab/sv/gitlab-kas/log/config] action create (up to date)
971 * ruby_block[verify_chown_persisted_on_gitlab-kas] action nothing (skipped due to action :nothing)
972 * link[/var/log/gitlab/gitlab-kas/config] action create (up to date)
973 * template[/opt/gitlab/sv/gitlab-kas/log/run] action create (up to date)
974 * directory[/opt/gitlab/sv/gitlab-kas/env] action create (up to date)
975 * ruby_block[Delete unmanaged env files for gitlab-kas service] action run (skipped due to only_if)
976 * template[/opt/gitlab/sv/gitlab-kas/check] action create (skipped due to only_if)
977 * template[/opt/gitlab/sv/gitlab-kas/finish] action create (skipped due to only_if)
978 * directory[/opt/gitlab/sv/gitlab-kas/control] action create (up to date)
979 * link[/opt/gitlab/init/gitlab-kas] action create (up to date)
980 * file[/opt/gitlab/sv/gitlab-kas/down] action nothing (skipped due to action :nothing)
981 * directory[/opt/gitlab/service] action create (up to date)
982 * link[/opt/gitlab/service/gitlab-kas] action create (up to date)
983 * ruby_block[wait for gitlab-kas service socket] action run (skipped due to not_if)
984 * file[/var/log/gitlab/gitlab-kas/current] action touch (skipped due to only_if)
985[2023-07-20T03:20:38+00:00] INFO: ruby_block[restart_log_service] called
986 - execute the ruby block restart_log_service
987 * directory[/opt/gitlab/service] action create (up to date)
988 * link[/opt/gitlab/service/gitlab-kas] action create (up to date)
989 * ruby_block[wait for gitlab-kas service socket] action run (skipped due to not_if)
990 * file[/var/log/gitlab/gitlab-kas/current] action touch (skipped due to only_if)
991
992Recipe: gitlab::database_migrations
993 * ruby_block[check remote PG version] action nothing (skipped due to action :nothing)
994 * rails_migration[gitlab-rails] action run (skipped due to only_if)
995Recipe: crond::disable
996 * service[crond] action nothing (skipped due to action :nothing)
997 * runit_service[crond] action disable
998 * ruby_block[disable crond] action run (skipped due to only_if)
999 (up to date)
1000Recipe: gitlab::puma_disable
1001 * service[puma] action nothing (skipped due to action :nothing)
1002 * runit_service[puma] action disable
1003 * ruby_block[disable puma] action run (skipped due to only_if)
1004 (up to date)
1005 * consul_service[rails] action delete
1006 * file[/var/opt/gitlab/consul/config.d/rails-service.json] action delete (up to date)
1007 (up to date)
1008Recipe: gitlab::sidekiq_disable
1009 * service[sidekiq] action nothing (skipped due to action :nothing)
1010 * runit_service[sidekiq] action disable
1011 * ruby_block[disable sidekiq] action run (skipped due to only_if)
1012 (up to date)
1013 * consul_service[sidekiq] action delete
1014 * file[/var/opt/gitlab/consul/config.d/sidekiq-service.json] action delete (up to date)
1015 (up to date)
1016Recipe: gitlab::gitlab-workhorse_disable
1017 * service[gitlab-workhorse] action nothing (skipped due to action :nothing)
1018 * runit_service[gitlab-workhorse] action disable
1019 * ruby_block[disable gitlab-workhorse] action run (skipped due to only_if)
1020 (up to date)
1021 * consul_service[workhorse] action delete
1022 * file[/var/opt/gitlab/consul/config.d/workhorse-service.json] action delete (up to date)
1023 (up to date)
1024Recipe: gitlab::mailroom_disable
1025 * service[mailroom] action nothing (skipped due to action :nothing)
1026 * runit_service[mailroom] action disable
1027 * ruby_block[disable mailroom] action run (skipped due to only_if)
1028 (up to date)
1029Recipe: gitlab::nginx_disable
1030 * service[nginx] action nothing (skipped due to action :nothing)
1031 * runit_service[nginx] action disable
1032 * ruby_block[disable nginx] action run (skipped due to only_if)
1033 (up to date)
1034 * consul_service[nginx] action delete
1035 * file[/var/opt/gitlab/consul/config.d/nginx-service.json] action delete (up to date)
1036 (up to date)
1037Recipe: gitlab::remote-syslog_disable
1038 * service[remote-syslog] action nothing (skipped due to action :nothing)
1039 * runit_service[remote-syslog] action disable
1040 * ruby_block[disable remote-syslog] action run (skipped due to only_if)
1041 (up to date)
1042Recipe: gitlab::bootstrap
1043 * file[/var/opt/gitlab/bootstrapped] action create[2023-07-20T03:20:38+00:00] INFO: file[/var/opt/gitlab/bootstrapped] created file /var/opt/gitlab/bootstrapped
1044 - create new file /var/opt/gitlab/bootstrapped[2023-07-20T03:20:38+00:00] INFO: file[/var/opt/gitlab/bootstrapped] updated file contents /var/opt/gitlab/bootstrapped
1045 - update content in file /var/opt/gitlab/bootstrapped from none to 4ae00c
1046 --- /var/opt/gitlab/bootstrapped 2023-07-20 03:20:38.000000000 +0000
1047 +++ /var/opt/gitlab/.chef-bootstrapped20230720-37-ujlaqc 2023-07-20 03:20:38.000000000 +0000
1048 @@ -1 +1,2 @@
1049 +All your bootstraps are belong to Chef[2023-07-20T03:20:38+00:00] INFO: file[/var/opt/gitlab/bootstrapped] owner changed to 0
1050[2023-07-20T03:20:38+00:00] INFO: file[/var/opt/gitlab/bootstrapped] group changed to 0
1051[2023-07-20T03:20:38+00:00] INFO: file[/var/opt/gitlab/bootstrapped] mode changed to 600
1052 - change mode from '' to '0600'
1053 - change owner from '' to 'root'
1054 - change group from '' to 'root'
1055Recipe: gitlab::storage-check_disable
1056 * service[storage-check] action nothing (skipped due to action :nothing)
1057 * runit_service[storage-check] action disable
1058 * ruby_block[disable storage-check] action run (skipped due to only_if)
1059 (up to date)
1060Recipe: gitlab-pages::disable
1061 * service[gitlab-pages] action nothing (skipped due to action :nothing)
1062 * runit_service[gitlab-pages] action disable
1063 * ruby_block[disable gitlab-pages] action run (skipped due to only_if)
1064 (up to date)
1065Recipe: registry::disable
1066 * service[registry] action nothing (skipped due to action :nothing)
1067 * runit_service[registry] action disable
1068 * ruby_block[disable registry] action run (skipped due to only_if)
1069 (up to date)
1070Recipe: mattermost::disable
1071 * service[mattermost] action nothing (skipped due to action :nothing)
1072 * runit_service[mattermost] action disable
1073 * ruby_block[disable mattermost] action run (skipped due to only_if)
1074 (up to date)
1075Recipe: letsencrypt::disable
1076 * crond_job[letsencrypt-renew] action delete
1077 * file[/var/opt/gitlab/crond/letsencrypt-renew] action delete (up to date)
1078 (up to date)
1079Recipe: monitoring::node-exporter_disable
1080 * service[node-exporter] action nothing (skipped due to action :nothing)
1081 * runit_service[node-exporter] action disable
1082 * ruby_block[disable node-exporter] action run (skipped due to only_if)
1083 (up to date)
1084 * consul_service[node-exporter] action delete
1085 * file[/var/opt/gitlab/consul/config.d/node-exporter-service.json] action delete (up to date)
1086 (up to date)
1087Recipe: monitoring::gitlab-exporter_disable
1088 * service[gitlab-exporter] action nothing (skipped due to action :nothing)
1089 * runit_service[gitlab-exporter] action disable
1090 * ruby_block[disable gitlab-exporter] action run (skipped due to only_if)
1091 (up to date)
1092 * consul_service[gitlab-exporter] action delete
1093 * file[/var/opt/gitlab/consul/config.d/gitlab-exporter-service.json] action delete (up to date)
1094 (up to date)
1095Recipe: monitoring::redis-exporter_disable
1096 * service[redis-exporter] action nothing (skipped due to action :nothing)
1097 * runit_service[redis-exporter] action disable
1098 * ruby_block[disable redis-exporter] action run (skipped due to only_if)
1099 (up to date)
1100 * consul_service[redis-exporter] action delete
1101 * file[/var/opt/gitlab/consul/config.d/redis-exporer-service.json] action delete (up to date)
1102 (up to date)
1103Recipe: monitoring::user
1104 * account[Prometheus user and group] action create (up to date)
1105Recipe: monitoring::prometheus
1106 * directory[/var/opt/gitlab/prometheus] action create (up to date)
1107 * directory[/var/opt/gitlab/prometheus/rules] action create[2023-07-20T03:20:39+00:00] INFO: directory[/var/opt/gitlab/prometheus/rules] created directory /var/opt/gitlab/prometheus/rules
1108 - create new directory /var/opt/gitlab/prometheus/rules[2023-07-20T03:20:39+00:00] INFO: directory[/var/opt/gitlab/prometheus/rules] owner changed to 992
1109[2023-07-20T03:20:39+00:00] INFO: directory[/var/opt/gitlab/prometheus/rules] mode changed to 750
1110 - change mode from '' to '0750'
1111 - change owner from '' to 'gitlab-prometheus'
1112 * directory[/var/log/gitlab/prometheus] action create[2023-07-20T03:20:39+00:00] INFO: directory[/var/log/gitlab/prometheus] created directory /var/log/gitlab/prometheus
1113 - create new directory /var/log/gitlab/prometheus[2023-07-20T03:20:39+00:00] INFO: directory[/var/log/gitlab/prometheus] owner changed to 992
1114[2023-07-20T03:20:39+00:00] INFO: directory[/var/log/gitlab/prometheus] mode changed to 700
1115 - change mode from '' to '0700'
1116 - change owner from '' to 'gitlab-prometheus'
1117 * directory[/opt/gitlab/etc/prometheus/env] action create[2023-07-20T03:20:39+00:00] INFO: directory[/opt/gitlab/etc/prometheus/env] created directory /opt/gitlab/etc/prometheus/env
1118 - create new directory /opt/gitlab/etc/prometheus/env[2023-07-20T03:20:39+00:00] INFO: directory[/opt/gitlab/etc/prometheus/env] owner changed to 992
1119[2023-07-20T03:20:39+00:00] INFO: directory[/opt/gitlab/etc/prometheus/env] mode changed to 700
1120 - change mode from '' to '0700'
1121 - change owner from '' to 'gitlab-prometheus'
1122 * env_dir[/opt/gitlab/etc/prometheus/env] action create
1123 * directory[/opt/gitlab/etc/prometheus/env] action create (up to date)
1124 * file[/opt/gitlab/etc/prometheus/env/SSL_CERT_DIR] action create[2023-07-20T03:20:39+00:00] INFO: file[/opt/gitlab/etc/prometheus/env/SSL_CERT_DIR] created file /opt/gitlab/etc/prometheus/env/SSL_CERT_DIR
1125 - create new file /opt/gitlab/etc/prometheus/env/SSL_CERT_DIR[2023-07-20T03:20:39+00:00] INFO: file[/opt/gitlab/etc/prometheus/env/SSL_CERT_DIR] updated file contents /opt/gitlab/etc/prometheus/env/SSL_CERT_DIR
1126 - update content in file /opt/gitlab/etc/prometheus/env/SSL_CERT_DIR from none to 4f45cf
1127 - suppressed sensitive resource
1128
1129 * execute[reload prometheus] action nothing (skipped due to action :nothing)
1130 * file[Prometheus config] action create[2023-07-20T03:20:39+00:00] INFO: file[Prometheus config] created file /var/opt/gitlab/prometheus/prometheus.yml
1131 - create new file /var/opt/gitlab/prometheus/prometheus.yml[2023-07-20T03:20:39+00:00] INFO: file[Prometheus config] updated file contents /var/opt/gitlab/prometheus/prometheus.yml
1132 - update content in file /var/opt/gitlab/prometheus/prometheus.yml from none to 12d959
1133 --- /var/opt/gitlab/prometheus/prometheus.yml 2023-07-20 03:20:39.000000000 +0000
1134 +++ /var/opt/gitlab/prometheus/.chef-prometheus20230720-37-c40cc0.yml 2023-07-20 03:20:39.000000000 +0000
1135 @@ -1,115 +1,230 @@
1136 +---
1137 +global:
1138 + scrape_interval: 15s
1139 + scrape_timeout: 15s
1140 + external_labels: {}
1141 +remote_read: []
1142 +remote_write: []
1143 +rule_files:
1144 +- "/var/opt/gitlab/prometheus/rules/*.rules"
1145 +scrape_configs:
1146 +- job_name: prometheus
1147 + static_configs:
1148 + - targets:
1149 + - localhost:9090
1150 +- job_name: gitaly
1151 + static_configs:
1152 + - targets:
1153 + - 0.0.0.0:9236
1154 +- job_name: kubernetes-cadvisor
1155 + scheme: https
1156 + tls_config:
1157 + ca_file: "/var/run/secrets/kubernetes.io/serviceaccount/ca.crt"
1158 + insecure_skip_verify: true
1159 + bearer_token_file: "/var/run/secrets/kubernetes.io/serviceaccount/token"
1160 + kubernetes_sd_configs:
1161 + - role: node
1162 + api_server: https://kubernetes.default.svc:443
1163 + tls_config:
1164 + ca_file: "/var/run/secrets/kubernetes.io/serviceaccount/ca.crt"
1165 + bearer_token_file: "/var/run/secrets/kubernetes.io/serviceaccount/token"
1166 + relabel_configs:
1167 + - action: labelmap
1168 + regex: __meta_kubernetes_node_label_(.+)
1169 + - target_label: __address__
1170 + replacement: kubernetes.default.svc:443
1171 + - source_labels:
1172 + - __meta_kubernetes_node_name
1173 + regex: "(.+)"
1174 + target_label: __metrics_path__
1175 + replacement: "/api/v1/nodes/${1}/proxy/metrics/cadvisor"
1176 + metric_relabel_configs:
1177 + - source_labels:
1178 + - pod_name
1179 + target_label: environment
1180 + regex: "(.+)-.+-.+"
1181 +- job_name: kubernetes-nodes
1182 + scheme: https
1183 + tls_config:
1184 + ca_file: "/var/run/secrets/kubernetes.io/serviceaccount/ca.crt"
1185 + insecure_skip_verify: true
1186 + bearer_token_file: "/var/run/secrets/kubernetes.io/serviceaccount/token"
1187 + kubernetes_sd_configs:
1188 + - role: node
1189 + api_server: https://kubernetes.default.svc:443
1190 + tls_config:
1191 + ca_file: "/var/run/secrets/kubernetes.io/serviceaccount/ca.crt"
1192 + bearer_token_file: "/var/run/secrets/kubernetes.io/serviceaccount/token"
1193 + relabel_configs:
1194 + - action: labelmap
1195 + regex: __meta_kubernetes_node_label_(.+)
1196 + - target_label: __address__
1197 + replacement: kubernetes.default.svc:443
1198 + - source_labels:
1199 + - __meta_kubernetes_node_name
1200 + regex: "(.+)"
1201 + target_label: __metrics_path__
1202 + replacement: "/api/v1/nodes/${1}/proxy/metrics"
1203 + metric_relabel_configs:
1204 + - source_labels:
1205 + - pod_name
1206 + target_label: environment
1207 + regex: "(.+)-.+-.+"
1208 +- job_name: kubernetes-pods
1209 + tls_config:
1210 + ca_file: "/var/run/secrets/kubernetes.io/serviceaccount/ca.crt"
1211 + insecure_skip_verify: true
1212 + bearer_token_file: "/var/run/secrets/kubernetes.io/serviceaccount/token"
1213 + kubernetes_sd_configs:
1214 + - role: pod
1215 + api_server: https://kubernetes.default.svc:443
1216 + tls_config:
1217 + ca_file: "/var/run/secrets/kubernetes.io/serviceaccount/ca.crt"
1218 + bearer_token_file: "/var/run/secrets/kubernetes.io/serviceaccount/token"
1219 + relabel_configs:
1220 + - source_labels:
1221 + - __meta_kubernetes_pod_annotation_prometheus_io_scrape
1222 + action: keep
1223 + regex: 'true'
1224 + - source_labels:
1225 + - __meta_kubernetes_pod_annotation_prometheus_io_path
1226 + action: replace
1227 + target_label: __metrics_path__
1228 + regex: "(.+)"
1229 + - source_labels:
1230 + - __address__
1231 + - __meta_kubernetes_pod_annotation_prometheus_io_port
1232 + action: replace
1233 + regex: "([^:]+)(?::[0-9]+)?;([0-9]+)"
1234 + replacement: "$1:$2"
1235 + target_label: __address__
1236 + - action: labelmap
1237 + regex: __meta_kubernetes_pod_label_(.+)
1238 + - source_labels:
1239 + - __meta_kubernetes_namespace
1240 + action: replace
1241 + target_label: kubernetes_namespace
1242 + - source_labels:
1243 + - __meta_kubernetes_pod_name
1244 + action: replace
1245 + target_label: kubernetes_pod_name
1246 +alerting:
1247 + alertmanagers:
1248 + - static_configs:
1249 + - targets:
1250 + - localhost:9093[2023-07-20T03:20:39+00:00] INFO: file[Prometheus config] owner changed to 992
1251[2023-07-20T03:20:39+00:00] INFO: file[Prometheus config] mode changed to 644
1252 - change mode from '' to '0644'
1253 - change owner from '' to 'gitlab-prometheus'
1254 * service[prometheus] action nothing (skipped due to action :nothing)
1255 * runit_service[prometheus] action enable
1256 * ruby_block[restart_service] action nothing (skipped due to action :nothing)
1257 * ruby_block[restart_log_service] action nothing (skipped due to action :nothing)
1258 * ruby_block[reload_log_service] action nothing (skipped due to action :nothing)
1259 * directory[/opt/gitlab/sv/prometheus] action create[2023-07-20T03:20:39+00:00] INFO: directory[/opt/gitlab/sv/prometheus] created directory /opt/gitlab/sv/prometheus
1260 - create new directory /opt/gitlab/sv/prometheus[2023-07-20T03:20:39+00:00] INFO: directory[/opt/gitlab/sv/prometheus] owner changed to 0
1261[2023-07-20T03:20:39+00:00] INFO: directory[/opt/gitlab/sv/prometheus] group changed to 0
1262[2023-07-20T03:20:39+00:00] INFO: directory[/opt/gitlab/sv/prometheus] mode changed to 755
1263 - change mode from '' to '0755'
1264 - change owner from '' to 'root'
1265 - change group from '' to 'root'
1266 * template[/opt/gitlab/sv/prometheus/run] action create[2023-07-20T03:20:39+00:00] INFO: template[/opt/gitlab/sv/prometheus/run] created file /opt/gitlab/sv/prometheus/run
1267 - create new file /opt/gitlab/sv/prometheus/run[2023-07-20T03:20:39+00:00] INFO: template[/opt/gitlab/sv/prometheus/run] updated file contents /opt/gitlab/sv/prometheus/run
1268 - update content in file /opt/gitlab/sv/prometheus/run from none to 60f91b
1269 --- /opt/gitlab/sv/prometheus/run 2023-07-20 03:20:39.000000000 +0000
1270 +++ /opt/gitlab/sv/prometheus/.chef-run20230720-37-omtxgm 2023-07-20 03:20:39.000000000 +0000
1271 @@ -1,8 +1,16 @@
1272 +#!/bin/sh
1273 +exec 2>&1
1274 +
1275 +umask 077
1276 +exec chpst -P -e /opt/gitlab/etc/prometheus/env \
1277 + -U gitlab-prometheus:gitlab-prometheus \
1278 + -u gitlab-prometheus:gitlab-prometheus \
1279 + /opt/gitlab/embedded/bin/prometheus --web.listen-address=localhost:9090 --storage.tsdb.path=/var/opt/gitlab/prometheus/data --config.file=/var/opt/gitlab/prometheus/prometheus.yml[2023-07-20T03:20:39+00:00] INFO: template[/opt/gitlab/sv/prometheus/run] owner changed to 0
1280[2023-07-20T03:20:39+00:00] INFO: template[/opt/gitlab/sv/prometheus/run] group changed to 0
1281[2023-07-20T03:20:39+00:00] INFO: template[/opt/gitlab/sv/prometheus/run] mode changed to 755
1282 - change mode from '' to '0755'
1283 - change owner from '' to 'root'
1284 - change group from '' to 'root'
1285 * directory[/opt/gitlab/sv/prometheus/log] action create[2023-07-20T03:20:39+00:00] INFO: directory[/opt/gitlab/sv/prometheus/log] created directory /opt/gitlab/sv/prometheus/log
1286 - create new directory /opt/gitlab/sv/prometheus/log[2023-07-20T03:20:39+00:00] INFO: directory[/opt/gitlab/sv/prometheus/log] owner changed to 0
1287[2023-07-20T03:20:39+00:00] INFO: directory[/opt/gitlab/sv/prometheus/log] group changed to 0
1288[2023-07-20T03:20:39+00:00] INFO: directory[/opt/gitlab/sv/prometheus/log] mode changed to 755
1289 - change mode from '' to '0755'
1290 - change owner from '' to 'root'
1291 - change group from '' to 'root'
1292 * directory[/opt/gitlab/sv/prometheus/log/main] action create[2023-07-20T03:20:39+00:00] INFO: directory[/opt/gitlab/sv/prometheus/log/main] created directory /opt/gitlab/sv/prometheus/log/main
1293 - create new directory /opt/gitlab/sv/prometheus/log/main[2023-07-20T03:20:39+00:00] INFO: directory[/opt/gitlab/sv/prometheus/log/main] owner changed to 0
1294[2023-07-20T03:20:39+00:00] INFO: directory[/opt/gitlab/sv/prometheus/log/main] group changed to 0
1295[2023-07-20T03:20:39+00:00] INFO: directory[/opt/gitlab/sv/prometheus/log/main] mode changed to 755
1296 - change mode from '' to '0755'
1297 - change owner from '' to 'root'
1298 - change group from '' to 'root'
1299 * template[/opt/gitlab/sv/prometheus/log/config] action create[2023-07-20T03:20:39+00:00] INFO: template[/opt/gitlab/sv/prometheus/log/config] created file /opt/gitlab/sv/prometheus/log/config
1300 - create new file /opt/gitlab/sv/prometheus/log/config[2023-07-20T03:20:39+00:00] INFO: template[/opt/gitlab/sv/prometheus/log/config] updated file contents /opt/gitlab/sv/prometheus/log/config
1301 - update content in file /opt/gitlab/sv/prometheus/log/config from none to 623c00
1302 --- /opt/gitlab/sv/prometheus/log/config 2023-07-20 03:20:39.000000000 +0000
1303 +++ /opt/gitlab/sv/prometheus/log/.chef-config20230720-37-3ur5sx 2023-07-20 03:20:39.000000000 +0000
1304 @@ -1,6 +1,12 @@
1305 +s209715200
1306 +n30
1307 +t86400
1308 +!gzip
1309 +
1310 +[2023-07-20T03:20:39+00:00] INFO: template[/opt/gitlab/sv/prometheus/log/config] owner changed to 0
1311[2023-07-20T03:20:39+00:00] INFO: template[/opt/gitlab/sv/prometheus/log/config] group changed to 0
1312[2023-07-20T03:20:39+00:00] INFO: template[/opt/gitlab/sv/prometheus/log/config] mode changed to 644
1313 - change mode from '' to '0644'
1314 - change owner from '' to 'root'
1315 - change group from '' to 'root'
1316[2023-07-20T03:20:39+00:00] INFO: template[/opt/gitlab/sv/prometheus/log/config] sending create action to ruby_block[verify_chown_persisted_on_prometheus] (immediate)
1317 * ruby_block[verify_chown_persisted_on_prometheus] action create[2023-07-20T03:20:39+00:00] INFO: ruby_block[verify_chown_persisted_on_prometheus] called
1318 - execute the ruby block verify_chown_persisted_on_prometheus
1319 * ruby_block[verify_chown_persisted_on_prometheus] action nothing (skipped due to action :nothing)
1320 * link[/var/log/gitlab/prometheus/config] action create[2023-07-20T03:20:39+00:00] INFO: link[/var/log/gitlab/prometheus/config] created
1321 - create symlink at /var/log/gitlab/prometheus/config to /opt/gitlab/sv/prometheus/log/config
1322 * template[/opt/gitlab/sv/prometheus/log/run] action create[2023-07-20T03:20:39+00:00] INFO: template[/opt/gitlab/sv/prometheus/log/run] created file /opt/gitlab/sv/prometheus/log/run
1323 - create new file /opt/gitlab/sv/prometheus/log/run[2023-07-20T03:20:39+00:00] INFO: template[/opt/gitlab/sv/prometheus/log/run] updated file contents /opt/gitlab/sv/prometheus/log/run
1324 - update content in file /opt/gitlab/sv/prometheus/log/run from none to 8afdb7
1325 --- /opt/gitlab/sv/prometheus/log/run 2023-07-20 03:20:39.000000000 +0000
1326 +++ /opt/gitlab/sv/prometheus/log/.chef-run20230720-37-t8j2e9 2023-07-20 03:20:39.000000000 +0000
1327 @@ -1,5 +1,10 @@
1328 +#!/bin/sh
1329 +exec chpst -P \
1330 + -U root:root \
1331 + -u root:root \
1332 + svlogd -tt /var/log/gitlab/prometheus[2023-07-20T03:20:39+00:00] INFO: template[/opt/gitlab/sv/prometheus/log/run] owner changed to 0
1333[2023-07-20T03:20:39+00:00] INFO: template[/opt/gitlab/sv/prometheus/log/run] group changed to 0
1334[2023-07-20T03:20:39+00:00] INFO: template[/opt/gitlab/sv/prometheus/log/run] mode changed to 755
1335 - change mode from '' to '0755'
1336 - change owner from '' to 'root'
1337 - change group from '' to 'root'
1338 * directory[/opt/gitlab/sv/prometheus/env] action create[2023-07-20T03:20:39+00:00] INFO: directory[/opt/gitlab/sv/prometheus/env] created directory /opt/gitlab/sv/prometheus/env
1339 - create new directory /opt/gitlab/sv/prometheus/env[2023-07-20T03:20:39+00:00] INFO: directory[/opt/gitlab/sv/prometheus/env] owner changed to 0
1340[2023-07-20T03:20:39+00:00] INFO: directory[/opt/gitlab/sv/prometheus/env] group changed to 0
1341[2023-07-20T03:20:39+00:00] INFO: directory[/opt/gitlab/sv/prometheus/env] mode changed to 755
1342 - change mode from '' to '0755'
1343 - change owner from '' to 'root'
1344 - change group from '' to 'root'
1345 * ruby_block[Delete unmanaged env files for prometheus service] action run (skipped due to only_if)
1346 * template[/opt/gitlab/sv/prometheus/check] action create (skipped due to only_if)
1347 * template[/opt/gitlab/sv/prometheus/finish] action create (skipped due to only_if)
1348 * directory[/opt/gitlab/sv/prometheus/control] action create[2023-07-20T03:20:39+00:00] INFO: directory[/opt/gitlab/sv/prometheus/control] created directory /opt/gitlab/sv/prometheus/control
1349 - create new directory /opt/gitlab/sv/prometheus/control[2023-07-20T03:20:39+00:00] INFO: directory[/opt/gitlab/sv/prometheus/control] owner changed to 0
1350[2023-07-20T03:20:39+00:00] INFO: directory[/opt/gitlab/sv/prometheus/control] group changed to 0
1351[2023-07-20T03:20:39+00:00] INFO: directory[/opt/gitlab/sv/prometheus/control] mode changed to 755
1352 - change mode from '' to '0755'
1353 - change owner from '' to 'root'
1354 - change group from '' to 'root'
1355 * link[/opt/gitlab/init/prometheus] action create[2023-07-20T03:20:39+00:00] INFO: link[/opt/gitlab/init/prometheus] created
1356 - create symlink at /opt/gitlab/init/prometheus to /opt/gitlab/embedded/bin/sv
1357 * file[/opt/gitlab/sv/prometheus/down] action nothing (skipped due to action :nothing)
1358[2023-07-20T03:20:39+00:00] INFO: template[/opt/gitlab/sv/prometheus/run] sending run action to ruby_block[restart_service] (delayed)
1359 * ruby_block[restart_service] action run (skipped due to only_if)
1360[2023-07-20T03:20:39+00:00] INFO: template[/opt/gitlab/sv/prometheus/log/config] sending create action to ruby_block[reload_log_service] (delayed)
1361 * ruby_block[reload_log_service] action create
1362 * ruby_block[restart_service] action nothing (skipped due to action :nothing)
1363 * ruby_block[restart_log_service] action nothing (skipped due to action :nothing)
1364 * ruby_block[reload_log_service] action nothing (skipped due to action :nothing)
1365 * directory[/opt/gitlab/sv/prometheus] action create (up to date)
1366 * template[/opt/gitlab/sv/prometheus/run] action create (up to date)
1367 * directory[/opt/gitlab/sv/prometheus/log] action create (up to date)
1368 * directory[/opt/gitlab/sv/prometheus/log/main] action create (up to date)
1369 * template[/opt/gitlab/sv/prometheus/log/config] action create (up to date)
1370 * ruby_block[verify_chown_persisted_on_prometheus] action nothing (skipped due to action :nothing)
1371 * link[/var/log/gitlab/prometheus/config] action create (up to date)
1372 * template[/opt/gitlab/sv/prometheus/log/run] action create (up to date)
1373 * directory[/opt/gitlab/sv/prometheus/env] action create (up to date)
1374 * ruby_block[Delete unmanaged env files for prometheus service] action run (skipped due to only_if)
1375 * template[/opt/gitlab/sv/prometheus/check] action create (skipped due to only_if)
1376 * template[/opt/gitlab/sv/prometheus/finish] action create (skipped due to only_if)
1377 * directory[/opt/gitlab/sv/prometheus/control] action create (up to date)
1378 * link[/opt/gitlab/init/prometheus] action create (up to date)
1379 * file[/opt/gitlab/sv/prometheus/down] action nothing (skipped due to action :nothing)
1380 * directory[/opt/gitlab/service] action create (up to date)
1381 * link[/opt/gitlab/service/prometheus] action create[2023-07-20T03:20:39+00:00] INFO: link[/opt/gitlab/service/prometheus] created
1382 - create symlink at /opt/gitlab/service/prometheus to /opt/gitlab/sv/prometheus
1383 * ruby_block[wait for prometheus service socket] action run[2023-07-20T03:20:44+00:00] INFO: ruby_block[wait for prometheus service socket] called
1384 - execute the ruby block wait for prometheus service socket
1385 * file[/var/log/gitlab/prometheus/current] action touch (skipped due to only_if)
1386[2023-07-20T03:20:44+00:00] INFO: ruby_block[reload_log_service] called
1387 - execute the ruby block reload_log_service
1388[2023-07-20T03:20:44+00:00] INFO: template[/opt/gitlab/sv/prometheus/log/run] sending run action to ruby_block[restart_log_service] (delayed)
1389 * ruby_block[restart_log_service] action run
1390 * ruby_block[restart_service] action nothing (skipped due to action :nothing)
1391 * ruby_block[restart_log_service] action nothing (skipped due to action :nothing)
1392 * ruby_block[reload_log_service] action nothing (skipped due to action :nothing)
1393 * directory[/opt/gitlab/sv/prometheus] action create (up to date)
1394 * template[/opt/gitlab/sv/prometheus/run] action create (up to date)
1395 * directory[/opt/gitlab/sv/prometheus/log] action create (up to date)
1396 * directory[/opt/gitlab/sv/prometheus/log/main] action create (up to date)
1397 * template[/opt/gitlab/sv/prometheus/log/config] action create (up to date)
1398 * ruby_block[verify_chown_persisted_on_prometheus] action nothing (skipped due to action :nothing)
1399 * link[/var/log/gitlab/prometheus/config] action create (up to date)
1400 * template[/opt/gitlab/sv/prometheus/log/run] action create (up to date)
1401 * directory[/opt/gitlab/sv/prometheus/env] action create (up to date)
1402 * ruby_block[Delete unmanaged env files for prometheus service] action run (skipped due to only_if)
1403 * template[/opt/gitlab/sv/prometheus/check] action create (skipped due to only_if)
1404 * template[/opt/gitlab/sv/prometheus/finish] action create (skipped due to only_if)
1405 * directory[/opt/gitlab/sv/prometheus/control] action create (up to date)
1406 * link[/opt/gitlab/init/prometheus] action create (up to date)
1407 * file[/opt/gitlab/sv/prometheus/down] action nothing (skipped due to action :nothing)
1408 * directory[/opt/gitlab/service] action create (up to date)
1409 * link[/opt/gitlab/service/prometheus] action create (up to date)
1410 * ruby_block[wait for prometheus service socket] action run (skipped due to not_if)
1411 * file[/var/log/gitlab/prometheus/current] action touch (skipped due to only_if)
1412[2023-07-20T03:20:45+00:00] INFO: ruby_block[restart_log_service] called
1413 - execute the ruby block restart_log_service
1414 * directory[/opt/gitlab/service] action create (up to date)
1415 * link[/opt/gitlab/service/prometheus] action create (up to date)
1416 * ruby_block[wait for prometheus service socket] action run (skipped due to not_if)
1417 * file[/var/log/gitlab/prometheus/current] action touch (skipped due to only_if)
1418
1419 * consul_service[prometheus] action delete
1420 * file[/var/opt/gitlab/consul/config.d/prometheus-service.json] action delete (up to date)
1421 (up to date)
1422 * execute[/opt/gitlab/bin/gitlab-ctl start prometheus] action run
1423 [execute] ok: run: prometheus: (pid 669) 2s
1424[2023-07-20T03:20:46+00:00] INFO: execute[/opt/gitlab/bin/gitlab-ctl start prometheus] ran successfully
1425 - execute /opt/gitlab/bin/gitlab-ctl start prometheus
1426 * template[/var/opt/gitlab/prometheus/rules/gitlab.rules] action create[2023-07-20T03:20:46+00:00] INFO: template[/var/opt/gitlab/prometheus/rules/gitlab.rules] created file /var/opt/gitlab/prometheus/rules/gitlab.rules
1427 - create new file /var/opt/gitlab/prometheus/rules/gitlab.rules[2023-07-20T03:20:46+00:00] INFO: template[/var/opt/gitlab/prometheus/rules/gitlab.rules] updated file contents /var/opt/gitlab/prometheus/rules/gitlab.rules
1428 - update content in file /var/opt/gitlab/prometheus/rules/gitlab.rules from none to a55956
1429 --- /var/opt/gitlab/prometheus/rules/gitlab.rules 2023-07-20 03:20:46.000000000 +0000
1430 +++ /var/opt/gitlab/prometheus/rules/.chef-gitlab20230720-37-fsb361.rules 2023-07-20 03:20:46.000000000 +0000
1431 @@ -1,492 +1,984 @@
1432 +---
1433 +groups:
1434 +- name: GitLab
1435 + rules:
1436 + - record: instance:puma_utilization:ratio
1437 + expr: >
1438 + sum by (instance) (
1439 + puma_active_connections
1440 + ) /
1441 + sum by (instance) (
1442 + puma_max_threads
1443 + )
1444 + - record: job_route_method_code:gitlab_workhorse_http_request_duration_seconds_count:rate5m
1445 + expr: >
1446 + sum by (job, route, method, code) (
1447 + rate(gitlab_workhorse_http_request_duration_seconds_count[5m])
1448 + )
1449 +
1450 + - alert: ServiceDown
1451 + expr: avg_over_time(up[5m]) * 100 < 50
1452 + annotations:
1453 + description: The service {{ $labels.job }} instance {{ $labels.instance }} is
1454 + not responding for more than 50% of the time for 5 minutes.
1455 + summary: The service {{ $labels.job }} is not responding
1456 + - alert: RedisDown
1457 + expr: avg_over_time(redis_up[5m]) * 100 < 50
1458 + annotations:
1459 + description: The Redis service {{ $labels.job }} instance {{ $labels.instance
1460 + }} is not responding for more than 50% of the time for 5 minutes.
1461 + summary: The Redis service {{ $labels.job }} is not responding
1462 + - alert: PostgresDown
1463 + expr: avg_over_time(pg_up[5m]) * 100 < 50
1464 + annotations:
1465 + description: The Postgres service {{ $labels.job }} instance {{ $labels.instance
1466 + }} is not responding for more than 50% of the time for 5 minutes.
1467 + summary: The Postgres service {{ $labels.job }} is not responding
1468 + - alert: PumaQueueing
1469 + expr: avg_over_time(puma_queued_connections[30m]) > 1
1470 + annotations:
1471 + description: Puma instance {{ $labels.instance }} is queueing requests with
1472 + an average of {{ $value | printf "%.1f" }} over the last 30 minutes.
1473 + summary: Puma is queueing requests
1474 + - alert: HighPumaUtilization
1475 + expr: instance:puma_utilization:ratio * 100 > 90
1476 + for: 60m
1477 + annotations:
1478 + description: Puma instance {{ $labels.instance }} has more than 90% thread utilization ({{ $value | printf "%.1f" }}%) over the last 60 minutes.
1479 + summary: Puma is has high utilization
1480 + - alert: SidekiqJobsQueuing
1481 + expr: sum by (name) (sidekiq_queue_size) > 0
1482 + for: 60m
1483 + annotations:
1484 + summary: Sidekiq has jobs queued
1485 + description: Sidekiq queue {{ $labels.name }} has {{ $value }} jobs queued for 60 minutes.
1486 + - alert: HighgRPCResourceExhaustedRate
1487 + expr: >
1488 + sum without (grpc_code, grpc_method, grpc_service, grpc_type) (
1489 + rate(grpc_server_handled_total{grpc_code="ResourceExhausted"}[5m])
1490 + ) /
1491 + sum without (grpc_code, grpc_method, grpc_service, grpc_type) (
1492 + rate(grpc_server_handled_total[5m])
1493 + ) * 100 > 1
1494 + for: 60m
1495 + annotations:
1496 + summary: High gRPC ResourceExhausted error rate
1497 + description: Job {{ $labels.job }} instance {{ $labels.instance }} gRPC is returning more than 1% ({{ $value | printf "%.1f" }}%) ResourceExhausted errors over the last 60 minutes.
1498 + - alert: PostgresDatabaseDeadlocks
1499 + expr: increase(pg_stat_database_deadlocks[5m]) > 0
1500 + annotations:
1501 + summary: Postgres database has deadlocks
1502 + description: Postgres database {{ $labels.instance }} had {{ $value | printf "%d" }} deadlocks in the last 5 minutes.
1503 + - alert: PostgresDatabaseDeadlockCancels
1504 + expr: increase(pg_stat_database_deadlocks[5m]) > 0
1505 + annotations:
1506 + summary: Postgres database has queries canceled due to deadlocks
1507 + description: Postgres database {{ $labels.instance }} had {{ $value | printf "%d" }} queries canceled due to deadlocks in the last 5 minutes.
1508 + # Low-traffic - < 10 QPS (600 RPM)
1509 + - alert: WorkhorseHighErrorRate
1510 + expr: >
1511 + (
1512 + sum without (job, code) (
1513 + job_route_method_code:gitlab_workhorse_http_request_duration_seconds_count:rate5m{code=~"5.."}
1514 + ) /
1515 + sum without (job,code) (
1516 + job_route_method_code:gitlab_workhorse_http_request_duration_seconds_count:rate5m
1517 + ) < 10
1518 + ) * 100 > 50
1519 + annotations:
1520 + summary: Workhorse has high error rates
1521 + description: Workhorse route {{ $labels.route }} method {{ $labels.method }} has more than 50% errors ({{ $value | printf "%.1f" }}%) for the last 60 minutes.
1522 + # High-traffic - >= 10 QPS (600 RPM)
1523 + - alert: WorkhorseHighErrorRate
1524 + expr: >
1525 + (
1526 + sum without (job, code) (
1527 + job_route_method_code:gitlab_workhorse_http_request_duration_seconds_count:rate5m{code=~"5.."}
1528 + ) /
1529 + sum without (job,code) (
1530 + job_route_method_code:gitlab_workhorse_http_request_duration_seconds_count:rate5m
1531 + ) > 10
1532 + ) * 100 > 10
1533 + annotations:
1534 + summary: Workhorse has high error rates
1535 + description: Workhorse route {{ $labels.route }} method {{ $labels.method }} has more than 10% errors ({{ $value | printf "%.1f" }}%) for the last 60 minutes.
1536 +
1537 +###
1538 +# These metrics are top-level GitLab Service Level Indicators (SLIs). They can
1539 +# be used to monitor the overall health of a GitLab instance.
1540 +- name: Service Level Indicators
1541 + interval: 30s
1542 + rules:
1543 + # Service availability.
1544 + - record: gitlab_sli:job:availability:ratio
1545 + expr: >
1546 + avg by (job) (
1547 + avg_over_time(up[30s])
1548 + )
1549 + # Rails worker/thread capacity.
1550 + - record: gitlab_sli:rails_active_connections:avg30s
1551 + expr: >
1552 + sum(avg_over_time(puma_active_connections[30s]))
1553 + - record: gitlab_sli:rails_queued_connections:avg30s
1554 + expr: >
1555 + sum(avg_over_time(puma_queued_connections[30s]))
1556 + - record: gitlab_sli:rails_active_connections:max30s
1557 + expr: >
1558 + sum(max_over_time(puma_active_connections[30s]))
1559 + - record: gitlab_sli:rails_queued_connections:max30s
1560 + expr: >
1561 + sum(max_over_time(puma_queued_connections[30s]))
1562 + - record: gitlab_sli:rails_workers:avg30s
1563 + expr: >
1564 + count(avg_over_time(ruby_memory_bytes[30s])) or
1565 + sum(avg_over_time(puma_max_threads[30s]))
1566 + # Redis CPU use.
1567 + - record: gitlab_sli:redis_cpu_seconds:rate1m
1568 + expr: >
1569 + (sum(rate(redis_used_cpu_sys[1m])) + sum(rate(redis_used_cpu_user[1m]))) or
1570 + (sum(rate(redis_cpu_sys_seconds_total[1m])) + sum(rate(redis_cpu_user_seconds_total[1m])))
1571 + # Workhorse traffic.
1572 + - record: gitlab_sli:code_method_route:workhorse_http_request_count:rate1m
1573 + expr: >
1574 + sum by (code,method,route) (
1575 + rate(gitlab_workhorse_http_request_duration_seconds_count[1m])
1576 + )
1577 + - record: gitlab_sli:code_method_route:workhorse_http_request_duration_seconds:rate1m
1578 + expr: >
1579 + sum by (code,method,route) (
1580 + rate(gitlab_workhorse_http_request_duration_seconds_sum[1m])
1581 + )
1582 +# SLI - Apdex
1583 +- name: Service Level Indicators - Apdex
1584 + interval: 1m
1585 + rules:
1586 + # Gitaly goserver
1587 + # Satisfied -> 0.5 seconds
1588 + # Acceptable -> 1 seconds
1589 + - record: gitlab_sli:gitlab_component_apdex:ratio
1590 + labels:
1591 + job: gitaly
1592 + expr: >
1593 + (
1594 + sum(rate(grpc_server_handling_seconds_bucket{job="gitaly",grpc_type="unary",le="0.5",grpc_method!~"GarbageCollect|Fsck|RepackFull|RepackIncremental|CommitLanguages|CreateRepositoryFromURL|UserRebase|UserSquash|CreateFork|UserUpdateBranch|FindRemoteRepository|UserCherryPick|FetchRemote|UserRevert|FindRemoteRootRef"}[1m]))
1595 + +
1596 + sum(rate(grpc_server_handling_seconds_bucket{job="gitaly",grpc_type="unary",le="1",grpc_method!~"GarbageCollect|Fsck|RepackFull|RepackIncremental|CommitLanguages|CreateRepositoryFromURL|UserRebase|UserSquash|CreateFork|UserUpdateBranch|FindRemoteRepository|UserCherryPick|FetchRemote|UserRevert|FindRemoteRootRef"}[1m]))
1597 + )
1598 + /
1599 + 2 / sum(rate(grpc_server_handling_seconds_count{job="gitaly",grpc_type="unary",grpc_method!~"GarbageCollect|Fsck|RepackFull|RepackIncremental|CommitLanguages|CreateRepositoryFromURL|UserRebase|UserSquash|CreateFork|UserUpdateBranch|FindRemoteRepository|UserCherryPick|FetchRemote|UserRevert|FindRemoteRootRef"}[1m]))
1600 + # Sidekiq TODO: https://gitlab.com/gitlab-org/gitlab-foss/issues/56752
1601 + # - record: gitlab_sli:gitlab_component_apdex:ratio
1602 + # labels:
1603 + # job: gitlab-sidekiq
1604 + # expr: >
1605 + # (
1606 + # sum(rate(sidekiq_jobs_completion_time_seconds_bucket{le="25"}[1m]))
1607 + # +
1608 + # sum(rate(sidekiq_jobs_completion_time_seconds_bucket{le="50"}[1m]))
1609 + # )
1610 + # / 2
1611 + # /
1612 + # sum(rate(sidekiq_jobs_completion_time_seconds_count[1m]))
1613 +
1614 + # Workhorse
1615 + # Satisfied -> 1 seconds
1616 + # Acceptable -> 10 seconds
1617 + - record: gitlab_sli:gitlab_component_apdex:ratio
1618 + labels:
1619 + job: gitlab-workhorse
1620 + expr: >
1621 + (
1622 + sum(rate(gitlab_workhorse_http_request_duration_seconds_bucket{le="1",route!="^/([^/]+/){1,}[^/]+/uploads\\z",route!="^/api/v4/jobs/request\\z"}[1m]))
1623 + +
1624 + sum(rate(gitlab_workhorse_http_request_duration_seconds_bucket{le="10",route!="^/([^/]+/){1,}[^/]+/uploads\\z",route!="^/api/v4/jobs/request\\z"}[1m]))
1625 + )
1626 + /
1627 + 2 / sum(rate(gitlab_workhorse_http_request_duration_seconds_count{route!="^/([^/]+/){1,}[^/]+/uploads\\z",route!="^/api/v4/jobs/request\\z"}[1m]))
1628 +
1629 +# SLI - Errors
1630 +- name: Service Level Indicators - Errors
1631 + interval: 1m
1632 + rules:
1633 + # PostgreSQL
1634 + - record: gitlab_sli:gitlab_component_ops:rate
1635 + labels:
1636 + job: postgres
1637 + expr: >
1638 + sum(rate(pg_stat_database_xact_commit[1m])) +
1639 + sum(rate(pg_stat_database_xact_rollback[1m]))
1640 + - record: gitlab_sli:gitlab_component_errors:rate
1641 + labels:
1642 + job: postgres
1643 + expr: sum(rate(pg_stat_database_xact_rollback[1m]))
1644 + # Rails (Puma)
1645 + - record: gitlab_sli:gitlab_component_ops:rate
1646 + labels:
1647 + job: gitlab-rails
1648 + expr: >
1649 + sum(
1650 + rate(http_request_duration_seconds_count{job="gitlab-rails"}[1m])
1651 + )
1652 + - record: gitlab_sli:gitlab_component_errors:rate
1653 + labels:
1654 + job: gitlab-rails
1655 + expr: >
1656 + sum(
1657 + rate(http_request_duration_seconds_count{job="gitlab-rails",status=~"5.."}[1m])
1658 + )
1659 + # Sidekiq TODO: https://gitlab.com/gitlab-org/gitlab-foss/issues/56752
1660 + # - record: gitlab_sli:gitlab_component_ops:rate
1661 + # labels:
1662 + # job: gitlab-sidekiq
1663 + # expr: >
1664 + # sum (
1665 + # rate(sidekiq_jobs_started_total{job="gitlab-sidekiq"}[1m])
1666 + # )
1667 + # - record: gitlab_sli:gitlab_component_errors:rate
1668 + # labels:
1669 + # job: gitlab-sidekiq
1670 + # expr: >
1671 + # sum (
1672 + # rate(sidekiq_jobs_failed_total{job="gitlab-sidekiq"}[1m])
1673 + # )
1674 + # Workhorse
1675 + - record: gitlab_sli:gitlab_component_ops:rate
1676 + labels:
1677 + job: gitlab-workhorse
1678 + expr: >
1679 + sum(
1680 + rate(gitlab_workhorse_http_requests_total{job="gitlab-workhorse"}[1m])
1681 + )
1682 + - record: gitlab_sli:gitlab_component_errors:rate
1683 + labels:
1684 + job: gitlab-workhorse
1685 + expr: >
1686 + sum(
1687 + rate(gitlab_workhorse_http_requests_total{job="gitlab-workhorse",code=~"5.."}[1m])
1688 + )
1689 + ###
1690 + # Calculate service error ratios
1691 + - record: gitlab_sli:gitlab_component_errors:ratio
1692 + expr: >
1693 + gitlab_sli:gitlab_component_errors:rate /
1694 + gitlab_sli:gitlab_component_ops:rate
1695 +
1696 +- name: GitLab Saturation Ratios
1697 + interval: 1m
1698 + rules:
1699 + # type: *, component: cpu
1700 + # this measures average CPU across all the cores for the entire fleet for the given service
1701 + - record: gitlab_sli:gitlab_component_saturation:ratio
1702 + labels:
1703 + component: 'cpu'
1704 + expr: >
1705 + avg(1 - rate(node_cpu_seconds_total{mode="idle"}[1m]))
1706 +
1707 + # type: *, component: single_node_cpu
1708 + # this measures the maximum cpu availability across all the codes on a single server
1709 + # this can be helpful if CPU is not even distributed across the fleet.
1710 + - record: gitlab_sli:gitlab_component_saturation:ratio
1711 + labels:
1712 + component: 'single_node_cpu'
1713 + expr: >
1714 + max(
1715 + avg(1 - rate(node_cpu_seconds_total{mode="idle"}[1m]))
1716 + )
1717 +
1718 + - record: gitlab_sli:gitlab_component_saturation:ratio
1719 + labels:
1720 + component: 'disk_space'
1721 + expr: >
1722 + max(
1723 + (
1724 + (
1725 + node_filesystem_size_bytes{fstype=~"ext.|xfs|nfs.?"}
1726 + -
1727 + node_filesystem_free_bytes{fstype=~"ext.|xfs|nfs.?"}
1728 + )
1729 + /
1730 + node_filesystem_size_bytes{fstype=~"ext.|xfs|nfs.?"}
1731 + )
1732 + )
1733 +
1734 + - record: gitlab_sli:gitlab_component_saturation:ratio
1735 + labels:
1736 + component: 'memory'
1737 + expr: max (instance:node_memory_utilization:ratio)
1738 +
1739 + - record: gitlab_sli:gitlab_component_saturation:ratio
1740 + labels:
1741 + component: 'single_threaded_cpu'
1742 + expr: >
1743 + clamp_max(
1744 + max (
1745 + (rate(redis_cpu_user_seconds_total[1m]) + rate(redis_cpu_sys_seconds_total[1m])) or
1746 + (rate(redis_used_cpu_user[1m]) + rate(redis_used_cpu_sys[1m]))
1747 + ),
1748 + 1
1749 + )
1750 +
1751 + - record: gitlab_sli:gitlab_component_saturation:ratio
1752 + labels:
1753 + component: 'connection_pool'
1754 + expr: >
1755 + clamp_max(
1756 + max(
1757 + max_over_time(pgbouncer_pools_server_active_connections{user="gitlab"}[1m]) /
1758 + (
1759 + (
1760 + pgbouncer_pools_server_idle_connections{user="gitlab"} +
1761 + pgbouncer_pools_server_active_connections{user="gitlab"} +
1762 + pgbouncer_pools_server_testing_connections{user="gitlab"} +
1763 + pgbouncer_pools_server_used_connections{user="gitlab"} +
1764 + pgbouncer_pools_server_login_connections{user="gitlab"}
1765 + )
1766 + > 0
1767 + )
1768 + ), 1)
1769 +
1770 + # type: postgres-delayed, postgres-archive, patroni
1771 + - record: gitlab_sli:gitlab_component_saturation:ratio
1772 + labels:
1773 + component: 'active_db_connections'
1774 + expr: >
1775 + clamp_max(
1776 + max(
1777 + sum without(state) (pg_stat_activity_count{datname="gitlabhq_production", state!="idle"})
1778 + /
1779 + (sum without(state) (pg_stat_activity_count{datname="gitlabhq_production"}) > 0)
1780 + ), 1)
1781 +
1782 + # type: redis, redis-cache, component: redis_clients
1783 + # Records the saturation of redis client connections against a redis fleet
1784 + - record: gitlab_sli:gitlab_component_saturation:ratio
1785 + labels:
1786 + component: 'redis_clients'
1787 + expr: >
1788 + max(
1789 + max_over_time(redis_connected_clients[1m])
1790 + /
1791 + redis_config_maxclients
1792 + )
1793 +
1794 + # Aggregate over all components within a service using max
1795 + - record: gitlab_sli:gitlab_service_saturation:ratio
1796 + expr: >
1797 + max by (component) (gitlab_sli:gitlab_component_saturation:ratio)
1798 +
1799 + # Calculate the "sapdex" - the saturation apdex for the metric. 1 < less then soft limit, 0.5 < hard limit, else 0
1800 + - record: gitlab_sli:gitlab_component_saturation:ratio:sapdex
1801 + expr: >
1802 + clamp_min(gitlab_sli:gitlab_component_saturation:ratio <= on(component) group_left slo:max:soft:gitlab_sli:gitlab_component_saturation:ratio, 1)
1803 + or
1804 + clamp_min(clamp_max(gitlab_sli:gitlab_component_saturation:ratio > on(component) group_left slo:max:soft:gitlab_sli:gitlab_component_saturation:ratio, 0.5), 0.5)
1805 + or
1806 + clamp_max(gitlab_sli:gitlab_component_saturation:ratio > on(component) group_left slo:max:hard:gitlab_sli:gitlab_component_saturation:ratio, 0)
1807 +
1808 + # Open file descriptors
1809 + - record: gitlab_sli:gitlab_component_saturation:ratio
1810 + labels:
1811 + component: 'open_fds'
1812 + expr: >
1813 + max(
1814 + max_over_time(process_open_fds[1m])
1815 + /
1816 + max_over_time(process_max_fds[1m])
1817 + )
1818 + - record: gitlab_sli:gitlab_component_saturation:ratio
1819 + labels:
1820 + component: 'open_ruby_fds'
1821 + expr: >
1822 + max(
1823 + max_over_time(ruby_file_descriptors[1m])
1824 + /
1825 + max_over_time(ruby_process_max_fds[1m])
1826 + )
1827 +
1828 +# Unlike other service metrics, we record the stats for each component independently
1829 +- name: GitLab Saturation Ratios Stats
1830 + interval: 5m
1831 + rules:
1832 + # Average values for each service, over a week
1833 + - record: gitlab_sli:gitlab_component_saturation:ratio:avg_over_time_1w
1834 + expr: >
1835 + avg_over_time(gitlab_sli:gitlab_component_saturation:ratio[1w])
1836 +
1837 + # Using linear week-on-week growth, what prediction to we have for 2w from now?
1838 + - record: gitlab_sli:gitlab_component_saturation:ratio:predict_linear_2w
1839 + expr: >
1840 + predict_linear(gitlab_sli:gitlab_component_saturation:ratio:avg_over_time_1w[1w], 86400 * 14)
1841 +
1842 + # Using linear week-on-week growth, what prediction to we have for 30d from now?
1843 + - record: gitlab_sli:gitlab_component_saturation:ratio:predict_linear_30d
1844 + expr: >
1845 + predict_linear(gitlab_sli:gitlab_component_saturation:ratio:avg_over_time_1w[1w], 86400 * 30)
1846 +
1847 + # Sapdex, average for week
1848 + - record: gitlab_sli:gitlab_component_saturation:ratio:sapdex:avg_over_time_1w
1849 + expr: >
1850 + avg_over_time(gitlab_sli:gitlab_component_saturation:ratio:sapdex[1w])
1851 +
1852 + # Sapdex long term trend forecasting
1853 + # Using linear week-on-week growth, what prediction to we have for 30d from now?
1854 + - record: gitlab_sli:gitlab_component_saturation:ratio:sapdex:avg_over_time_1w:predict_linear_30d
1855 + expr: >
1856 + predict_linear(gitlab_sli:gitlab_component_saturation:ratio:sapdex:avg_over_time_1w[1w], 86400 * 30)
1857 +
1858 +###
1859 +# Metrics used by Usage Ping reports. These are primarily for self-managed installations.
1860 +# See https://docs.gitlab.com/ee/development/telemetry/usage_ping.html#prometheus-queries
1861 +#
1862 +- name: GitLab Usage Ping
1863 + interval: 5m
1864 + rules:
1865 + ###
1866 + # Application wide metrics
1867 + #
1868 + - record: gitlab_usage_ping:ops:rate5m
1869 + labels:
1870 + service: workhorse
1871 + component: http_requests
1872 + expr: sum (rate (gitlab_workhorse_http_requests_total[5m]))
1873 +
1874 + # Database Primary Performance Indicator: Query apdex
1875 + # Target -> 100ms
1876 + # Tolerable -> 250ms
1877 + - record: gitlab_usage_ping:sql_duration_apdex:ratio_rate5m
1878 + expr: >
1879 + (
1880 + sum(rate(gitlab_sql_duration_seconds_bucket{le="0.1"}[5m]))
1881 + +
1882 + sum(rate(gitlab_sql_duration_seconds_bucket{le="0.25"}[5m]))
1883 + )
1884 + /
1885 + 2
1886 + /
1887 + (
1888 + sum(rate(gitlab_sql_duration_seconds_count[5m])) > 0
1889 + )
1890 +
1891 + - record: gitlab_usage_ping:gitaly_apdex:ratio_avg_over_time_5m
1892 + expr: avg_over_time(gitlab_sli:gitlab_component_apdex:ratio{job="gitaly"}[5m])
1893 +
1894 + ###
1895 + # Node level metrics
1896 + #
1897 + - record: gitlab_usage_ping:node_cpus:count
1898 + expr: max (instance:node_cpus:count) by (instance)
1899 + - record: gitlab_usage_ping:node_memory_total_bytes:max
1900 + expr: max (node_memory_MemTotal_bytes) by (instance)
1901 + - record: gitlab_usage_ping:node_cpu_utilization:avg
1902 + expr: avg (instance:node_cpu_utilization:ratio) by (instance)
1903 + - record: gitlab_usage_ping:node_memory_utilization:avg
1904 + expr: avg (instance:node_memory_utilization:ratio) by (instance)
1905 +
1906 + ###
1907 + # Service level metrics
1908 + #
1909 + - record: gitlab_usage_ping:node_service_process_resident_memory_bytes:avg
1910 + expr: avg by (instance, job) ({__name__ =~ "(ruby_){0,1}process_resident_memory_bytes", job != "gitlab_exporter_process"})
1911 + - record: gitlab_usage_ping:node_service_process_unique_memory_bytes:avg
1912 + expr: avg by (instance, job) ({__name__ =~ "(ruby_){0,1}process_unique_memory_bytes", job != "gitlab_exporter_process"})
1913 + - record: gitlab_usage_ping:node_service_process_proportional_memory_bytes:avg
1914 + expr: avg by (instance, job) ({__name__ =~ "(ruby_){0,1}process_proportional_memory_bytes", job != "gitlab_exporter_process"})
1915 + - record: gitlab_usage_ping:node_service_process:count
1916 + expr: >
1917 + count by (instance, job) (
1918 + {__name__ =~ "(ruby_){0,1}process_start_time_seconds", job != "gitlab_exporter_process"}
1919 + )
1920 + - record: gitlab_usage_ping:node_service_app_server_workers:sum
1921 + labels:
1922 + server: puma
1923 + expr: sum by (instance, job) (puma_workers)[2023-07-20T03:20:46+00:00] INFO: template[/var/opt/gitlab/prometheus/rules/gitlab.rules] owner changed to 992
1924[2023-07-20T03:20:46+00:00] INFO: template[/var/opt/gitlab/prometheus/rules/gitlab.rules] mode changed to 644
1925 - change mode from '' to '0644'
1926 - change owner from '' to 'gitlab-prometheus'
1927[2023-07-20T03:20:46+00:00] INFO: template[/var/opt/gitlab/prometheus/rules/gitlab.rules] not queuing delayed action run on execute[reload prometheus] (delayed), as it's already been queued
1928 * template[/var/opt/gitlab/prometheus/rules/node.rules] action create[2023-07-20T03:20:46+00:00] INFO: template[/var/opt/gitlab/prometheus/rules/node.rules] created file /var/opt/gitlab/prometheus/rules/node.rules
1929 - create new file /var/opt/gitlab/prometheus/rules/node.rules[2023-07-20T03:20:46+00:00] INFO: template[/var/opt/gitlab/prometheus/rules/node.rules] updated file contents /var/opt/gitlab/prometheus/rules/node.rules
1930 - update content in file /var/opt/gitlab/prometheus/rules/node.rules from none to bea78d
1931 --- /var/opt/gitlab/prometheus/rules/node.rules 2023-07-20 03:20:46.000000000 +0000
1932 +++ /var/opt/gitlab/prometheus/rules/.chef-node20230720-37-yvwyhs.rules 2023-07-20 03:20:46.000000000 +0000
1933 @@ -1,42 +1,84 @@
1934 +groups:
1935 +- name: Node
1936 + rules:
1937 + - record: instance:node_cpus:count
1938 + expr: count without(cpu, mode) (node_cpu{mode="idle"})
1939 + - record: instance:node_cpus:count
1940 + expr: count without(cpu, mode) (node_cpu_seconds_total{mode="idle"})
1941 + - record: instance_cpu:node_cpu_seconds_not_idle:rate5m
1942 + expr: sum without(mode) (rate(node_cpu{mode!="idle"}[5m]))
1943 + - record: instance_cpu:node_cpu_seconds_not_idle:rate5m
1944 + expr: sum without(mode) (rate(node_cpu_seconds_total{mode!="idle"}[5m]))
1945 + - record: instance_mode:node_cpu_seconds:rate5m
1946 + expr: sum without(cpu) (rate(node_cpu[5m]))
1947 + - record: instance_mode:node_cpu_seconds:rate5m
1948 + expr: sum without(cpu) (rate(node_cpu_seconds_total[5m]))
1949 + - record: instance:node_cpu_utilization:ratio
1950 + expr: sum without(mode) (instance_mode:node_cpu_seconds:rate5m{mode!="idle"})
1951 + / instance:node_cpus:count
1952 + - record: instance:node_memory_utilization:ratio
1953 + expr: >
1954 + 1 - (
1955 + node_memory_MemAvailable_bytes or
1956 + (node_memory_MemFree_bytes + node_memory_Buffers_bytes + node_memory_Cached_bytes)
1957 + ) / node_memory_MemTotal_bytes
1958 + - record: instance:node_filesystem_avail:ratio
1959 + expr: node_filesystem_avail_bytes / (node_filesystem_size_bytes > 0)
1960 + - alert: FilesystemAlmostFull
1961 + expr: instance:node_filesystem_avail:ratio * 100 < 5 and node_filesystem_avail_bytes < 10 * 1024 * 1024 * 1024
1962 + for: 10m
1963 + annotations:
1964 + description: The filesystem {{ $labels.device }}:{{ $labels.mountpoint }} on
1965 + {{ $labels.instance }} has {{ $value | printf "%.2f" }}% space available.
1966 + summary: The filesystem {{ $labels.device }}:{{ $labels.mountpoint }} is almost
1967 + full
1968 + - alert: FilesystemFullIn1Day
1969 + expr: predict_linear(node_filesystem_avail_bytes[6h], 24 * 3600) < 0
1970 + for: 30m
1971 + annotations:
1972 + description: The filesystem {{ $labels.device }}:{{ $labels.mountpoint }} on
1973 + {{ $labels.instance }} will be full in the next 24 hours.
1974 + summary: The filesystem {{ $labels.device }}:{{ $labels.mountpoint }} will be
1975 + full within 24 hours[2023-07-20T03:20:46+00:00] INFO: template[/var/opt/gitlab/prometheus/rules/node.rules] owner changed to 992
1976[2023-07-20T03:20:46+00:00] INFO: template[/var/opt/gitlab/prometheus/rules/node.rules] mode changed to 644
1977 - change mode from '' to '0644'
1978 - change owner from '' to 'gitlab-prometheus'
1979[2023-07-20T03:20:46+00:00] INFO: template[/var/opt/gitlab/prometheus/rules/node.rules] not queuing delayed action run on execute[reload prometheus] (delayed), as it's already been queued
1980Recipe: monitoring::alertmanager
1981 * directory[/var/opt/gitlab/alertmanager] action create[2023-07-20T03:20:46+00:00] INFO: directory[/var/opt/gitlab/alertmanager] created directory /var/opt/gitlab/alertmanager
1982 - create new directory /var/opt/gitlab/alertmanager[2023-07-20T03:20:46+00:00] INFO: directory[/var/opt/gitlab/alertmanager] owner changed to 992
1983[2023-07-20T03:20:46+00:00] INFO: directory[/var/opt/gitlab/alertmanager] mode changed to 750
1984 - change mode from '' to '0750'
1985 - change owner from '' to 'gitlab-prometheus'
1986 * directory[/var/log/gitlab/alertmanager] action create[2023-07-20T03:20:46+00:00] INFO: directory[/var/log/gitlab/alertmanager] created directory /var/log/gitlab/alertmanager
1987 - create new directory /var/log/gitlab/alertmanager[2023-07-20T03:20:46+00:00] INFO: directory[/var/log/gitlab/alertmanager] owner changed to 992
1988[2023-07-20T03:20:46+00:00] INFO: directory[/var/log/gitlab/alertmanager] mode changed to 700
1989 - change mode from '' to '0700'
1990 - change owner from '' to 'gitlab-prometheus'
1991 * directory[/opt/gitlab/etc/alertmanager/env] action create[2023-07-20T03:20:46+00:00] INFO: directory[/opt/gitlab/etc/alertmanager/env] created directory /opt/gitlab/etc/alertmanager/env
1992 - create new directory /opt/gitlab/etc/alertmanager/env[2023-07-20T03:20:46+00:00] INFO: directory[/opt/gitlab/etc/alertmanager/env] owner changed to 992
1993[2023-07-20T03:20:46+00:00] INFO: directory[/opt/gitlab/etc/alertmanager/env] mode changed to 700
1994 - change mode from '' to '0700'
1995 - change owner from '' to 'gitlab-prometheus'
1996 * env_dir[/opt/gitlab/etc/alertmanager/env] action create
1997 * directory[/opt/gitlab/etc/alertmanager/env] action create (up to date)
1998 * file[/opt/gitlab/etc/alertmanager/env/SSL_CERT_DIR] action create[2023-07-20T03:20:46+00:00] INFO: file[/opt/gitlab/etc/alertmanager/env/SSL_CERT_DIR] created file /opt/gitlab/etc/alertmanager/env/SSL_CERT_DIR
1999 - create new file /opt/gitlab/etc/alertmanager/env/SSL_CERT_DIR[2023-07-20T03:20:46+00:00] INFO: file[/opt/gitlab/etc/alertmanager/env/SSL_CERT_DIR] updated file contents /opt/gitlab/etc/alertmanager/env/SSL_CERT_DIR
2000 - update content in file /opt/gitlab/etc/alertmanager/env/SSL_CERT_DIR from none to 4f45cf
2001 - suppressed sensitive resource
2002
2003 * file[Alertmanager config] action create[2023-07-20T03:20:46+00:00] INFO: file[Alertmanager config] created file /var/opt/gitlab/alertmanager/alertmanager.yml
2004 - create new file /var/opt/gitlab/alertmanager/alertmanager.yml[2023-07-20T03:20:46+00:00] INFO: file[Alertmanager config] updated file contents /var/opt/gitlab/alertmanager/alertmanager.yml
2005 - update content in file /var/opt/gitlab/alertmanager/alertmanager.yml from none to 21b7be
2006 --- /var/opt/gitlab/alertmanager/alertmanager.yml 2023-07-20 03:20:46.000000000 +0000
2007 +++ /var/opt/gitlab/alertmanager/.chef-alertmanager20230720-37-e9jhp6.yml 2023-07-20 03:20:46.000000000 +0000
2008 @@ -1,9 +1,18 @@
2009 +---
2010 +global: {}
2011 +templates: []
2012 +route:
2013 + receiver: default-receiver
2014 + routes: []
2015 +receivers:
2016 +- name: default-receiver
2017 +inhibit_rules: [][2023-07-20T03:20:46+00:00] INFO: file[Alertmanager config] owner changed to 992
2018[2023-07-20T03:20:46+00:00] INFO: file[Alertmanager config] mode changed to 644
2019 - change mode from '' to '0644'
2020 - change owner from '' to 'gitlab-prometheus'
2021[2023-07-20T03:20:46+00:00] INFO: file[Alertmanager config] not queuing delayed action restart on runit_service[alertmanager] (delayed), as it's already been queued
2022 * service[alertmanager] action nothing (skipped due to action :nothing)
2023 * runit_service[alertmanager] action enable
2024 * ruby_block[restart_service] action nothing (skipped due to action :nothing)
2025 * ruby_block[restart_log_service] action nothing (skipped due to action :nothing)
2026 * ruby_block[reload_log_service] action nothing (skipped due to action :nothing)
2027 * directory[/opt/gitlab/sv/alertmanager] action create[2023-07-20T03:20:46+00:00] INFO: directory[/opt/gitlab/sv/alertmanager] created directory /opt/gitlab/sv/alertmanager
2028 - create new directory /opt/gitlab/sv/alertmanager[2023-07-20T03:20:46+00:00] INFO: directory[/opt/gitlab/sv/alertmanager] owner changed to 0
2029[2023-07-20T03:20:46+00:00] INFO: directory[/opt/gitlab/sv/alertmanager] group changed to 0
2030[2023-07-20T03:20:46+00:00] INFO: directory[/opt/gitlab/sv/alertmanager] mode changed to 755
2031 - change mode from '' to '0755'
2032 - change owner from '' to 'root'
2033 - change group from '' to 'root'
2034 * template[/opt/gitlab/sv/alertmanager/run] action create[2023-07-20T03:20:46+00:00] INFO: template[/opt/gitlab/sv/alertmanager/run] created file /opt/gitlab/sv/alertmanager/run
2035 - create new file /opt/gitlab/sv/alertmanager/run[2023-07-20T03:20:46+00:00] INFO: template[/opt/gitlab/sv/alertmanager/run] updated file contents /opt/gitlab/sv/alertmanager/run
2036 - update content in file /opt/gitlab/sv/alertmanager/run from none to 36da8b
2037 --- /opt/gitlab/sv/alertmanager/run 2023-07-20 03:20:46.000000000 +0000
2038 +++ /opt/gitlab/sv/alertmanager/.chef-run20230720-37-mazb3l 2023-07-20 03:20:46.000000000 +0000
2039 @@ -1,8 +1,16 @@
2040 +#!/bin/sh
2041 +exec 2>&1
2042 +
2043 +umask 077
2044 +exec chpst -P -e /opt/gitlab/etc/alertmanager/env \
2045 + -U gitlab-prometheus:gitlab-prometheus \
2046 + -u gitlab-prometheus:gitlab-prometheus \
2047 + /opt/gitlab/embedded/bin/alertmanager --web.listen-address=localhost:9093 --storage.path=/var/opt/gitlab/alertmanager/data --config.file=/var/opt/gitlab/alertmanager/alertmanager.yml[2023-07-20T03:20:46+00:00] INFO: template[/opt/gitlab/sv/alertmanager/run] owner changed to 0
2048[2023-07-20T03:20:46+00:00] INFO: template[/opt/gitlab/sv/alertmanager/run] group changed to 0
2049[2023-07-20T03:20:46+00:00] INFO: template[/opt/gitlab/sv/alertmanager/run] mode changed to 755
2050 - change mode from '' to '0755'
2051 - change owner from '' to 'root'
2052 - change group from '' to 'root'
2053 * directory[/opt/gitlab/sv/alertmanager/log] action create[2023-07-20T03:20:46+00:00] INFO: directory[/opt/gitlab/sv/alertmanager/log] created directory /opt/gitlab/sv/alertmanager/log
2054 - create new directory /opt/gitlab/sv/alertmanager/log[2023-07-20T03:20:46+00:00] INFO: directory[/opt/gitlab/sv/alertmanager/log] owner changed to 0
2055[2023-07-20T03:20:46+00:00] INFO: directory[/opt/gitlab/sv/alertmanager/log] group changed to 0
2056[2023-07-20T03:20:46+00:00] INFO: directory[/opt/gitlab/sv/alertmanager/log] mode changed to 755
2057 - change mode from '' to '0755'
2058 - change owner from '' to 'root'
2059 - change group from '' to 'root'
2060 * directory[/opt/gitlab/sv/alertmanager/log/main] action create[2023-07-20T03:20:46+00:00] INFO: directory[/opt/gitlab/sv/alertmanager/log/main] created directory /opt/gitlab/sv/alertmanager/log/main
2061 - create new directory /opt/gitlab/sv/alertmanager/log/main[2023-07-20T03:20:46+00:00] INFO: directory[/opt/gitlab/sv/alertmanager/log/main] owner changed to 0
2062[2023-07-20T03:20:46+00:00] INFO: directory[/opt/gitlab/sv/alertmanager/log/main] group changed to 0
2063[2023-07-20T03:20:46+00:00] INFO: directory[/opt/gitlab/sv/alertmanager/log/main] mode changed to 755
2064 - change mode from '' to '0755'
2065 - change owner from '' to 'root'
2066 - change group from '' to 'root'
2067 * template[/opt/gitlab/sv/alertmanager/log/config] action create[2023-07-20T03:20:46+00:00] INFO: template[/opt/gitlab/sv/alertmanager/log/config] created file /opt/gitlab/sv/alertmanager/log/config
2068 - create new file /opt/gitlab/sv/alertmanager/log/config[2023-07-20T03:20:46+00:00] INFO: template[/opt/gitlab/sv/alertmanager/log/config] updated file contents /opt/gitlab/sv/alertmanager/log/config
2069 - update content in file /opt/gitlab/sv/alertmanager/log/config from none to 623c00
2070 --- /opt/gitlab/sv/alertmanager/log/config 2023-07-20 03:20:46.000000000 +0000
2071 +++ /opt/gitlab/sv/alertmanager/log/.chef-config20230720-37-sze46u 2023-07-20 03:20:46.000000000 +0000
2072 @@ -1,6 +1,12 @@
2073 +s209715200
2074 +n30
2075 +t86400
2076 +!gzip
2077 +
2078 +[2023-07-20T03:20:46+00:00] INFO: template[/opt/gitlab/sv/alertmanager/log/config] owner changed to 0
2079[2023-07-20T03:20:46+00:00] INFO: template[/opt/gitlab/sv/alertmanager/log/config] group changed to 0
2080[2023-07-20T03:20:46+00:00] INFO: template[/opt/gitlab/sv/alertmanager/log/config] mode changed to 644
2081 - change mode from '' to '0644'
2082 - change owner from '' to 'root'
2083 - change group from '' to 'root'
2084[2023-07-20T03:20:46+00:00] INFO: template[/opt/gitlab/sv/alertmanager/log/config] sending create action to ruby_block[verify_chown_persisted_on_alertmanager] (immediate)
2085 * ruby_block[verify_chown_persisted_on_alertmanager] action create[2023-07-20T03:20:46+00:00] INFO: ruby_block[verify_chown_persisted_on_alertmanager] called
2086 - execute the ruby block verify_chown_persisted_on_alertmanager
2087 * ruby_block[verify_chown_persisted_on_alertmanager] action nothing (skipped due to action :nothing)
2088 * link[/var/log/gitlab/alertmanager/config] action create[2023-07-20T03:20:46+00:00] INFO: link[/var/log/gitlab/alertmanager/config] created
2089 - create symlink at /var/log/gitlab/alertmanager/config to /opt/gitlab/sv/alertmanager/log/config
2090 * template[/opt/gitlab/sv/alertmanager/log/run] action create[2023-07-20T03:20:46+00:00] INFO: template[/opt/gitlab/sv/alertmanager/log/run] created file /opt/gitlab/sv/alertmanager/log/run
2091 - create new file /opt/gitlab/sv/alertmanager/log/run[2023-07-20T03:20:46+00:00] INFO: template[/opt/gitlab/sv/alertmanager/log/run] updated file contents /opt/gitlab/sv/alertmanager/log/run
2092 - update content in file /opt/gitlab/sv/alertmanager/log/run from none to 9a4626
2093 --- /opt/gitlab/sv/alertmanager/log/run 2023-07-20 03:20:46.000000000 +0000
2094 +++ /opt/gitlab/sv/alertmanager/log/.chef-run20230720-37-darxx8 2023-07-20 03:20:46.000000000 +0000
2095 @@ -1,5 +1,10 @@
2096 +#!/bin/sh
2097 +exec chpst -P \
2098 + -U root:root \
2099 + -u root:root \
2100 + svlogd -tt /var/log/gitlab/alertmanager[2023-07-20T03:20:46+00:00] INFO: template[/opt/gitlab/sv/alertmanager/log/run] owner changed to 0
2101[2023-07-20T03:20:46+00:00] INFO: template[/opt/gitlab/sv/alertmanager/log/run] group changed to 0
2102[2023-07-20T03:20:46+00:00] INFO: template[/opt/gitlab/sv/alertmanager/log/run] mode changed to 755
2103 - change mode from '' to '0755'
2104 - change owner from '' to 'root'
2105 - change group from '' to 'root'
2106 * directory[/opt/gitlab/sv/alertmanager/env] action create[2023-07-20T03:20:46+00:00] INFO: directory[/opt/gitlab/sv/alertmanager/env] created directory /opt/gitlab/sv/alertmanager/env
2107 - create new directory /opt/gitlab/sv/alertmanager/env[2023-07-20T03:20:46+00:00] INFO: directory[/opt/gitlab/sv/alertmanager/env] owner changed to 0
2108[2023-07-20T03:20:46+00:00] INFO: directory[/opt/gitlab/sv/alertmanager/env] group changed to 0
2109[2023-07-20T03:20:46+00:00] INFO: directory[/opt/gitlab/sv/alertmanager/env] mode changed to 755
2110 - change mode from '' to '0755'
2111 - change owner from '' to 'root'
2112 - change group from '' to 'root'
2113 * ruby_block[Delete unmanaged env files for alertmanager service] action run (skipped due to only_if)
2114 * template[/opt/gitlab/sv/alertmanager/check] action create (skipped due to only_if)
2115 * template[/opt/gitlab/sv/alertmanager/finish] action create (skipped due to only_if)
2116 * directory[/opt/gitlab/sv/alertmanager/control] action create[2023-07-20T03:20:46+00:00] INFO: directory[/opt/gitlab/sv/alertmanager/control] created directory /opt/gitlab/sv/alertmanager/control
2117 - create new directory /opt/gitlab/sv/alertmanager/control[2023-07-20T03:20:46+00:00] INFO: directory[/opt/gitlab/sv/alertmanager/control] owner changed to 0
2118[2023-07-20T03:20:46+00:00] INFO: directory[/opt/gitlab/sv/alertmanager/control] group changed to 0
2119[2023-07-20T03:20:46+00:00] INFO: directory[/opt/gitlab/sv/alertmanager/control] mode changed to 755
2120 - change mode from '' to '0755'
2121 - change owner from '' to 'root'
2122 - change group from '' to 'root'
2123 * link[/opt/gitlab/init/alertmanager] action create[2023-07-20T03:20:46+00:00] INFO: link[/opt/gitlab/init/alertmanager] created
2124 - create symlink at /opt/gitlab/init/alertmanager to /opt/gitlab/embedded/bin/sv
2125 * file[/opt/gitlab/sv/alertmanager/down] action nothing (skipped due to action :nothing)
2126[2023-07-20T03:20:46+00:00] INFO: template[/opt/gitlab/sv/alertmanager/run] sending run action to ruby_block[restart_service] (delayed)
2127 * ruby_block[restart_service] action run (skipped due to only_if)
2128[2023-07-20T03:20:46+00:00] INFO: template[/opt/gitlab/sv/alertmanager/log/config] sending create action to ruby_block[reload_log_service] (delayed)
2129 * ruby_block[reload_log_service] action create
2130 * ruby_block[restart_service] action nothing (skipped due to action :nothing)
2131 * ruby_block[restart_log_service] action nothing (skipped due to action :nothing)
2132 * ruby_block[reload_log_service] action nothing (skipped due to action :nothing)
2133 * directory[/opt/gitlab/sv/alertmanager] action create (up to date)
2134 * template[/opt/gitlab/sv/alertmanager/run] action create (up to date)
2135 * directory[/opt/gitlab/sv/alertmanager/log] action create (up to date)
2136 * directory[/opt/gitlab/sv/alertmanager/log/main] action create (up to date)
2137 * template[/opt/gitlab/sv/alertmanager/log/config] action create (up to date)
2138 * ruby_block[verify_chown_persisted_on_alertmanager] action nothing (skipped due to action :nothing)
2139 * link[/var/log/gitlab/alertmanager/config] action create (up to date)
2140 * template[/opt/gitlab/sv/alertmanager/log/run] action create (up to date)
2141 * directory[/opt/gitlab/sv/alertmanager/env] action create (up to date)
2142 * ruby_block[Delete unmanaged env files for alertmanager service] action run (skipped due to only_if)
2143 * template[/opt/gitlab/sv/alertmanager/check] action create (skipped due to only_if)
2144 * template[/opt/gitlab/sv/alertmanager/finish] action create (skipped due to only_if)
2145 * directory[/opt/gitlab/sv/alertmanager/control] action create (up to date)
2146 * link[/opt/gitlab/init/alertmanager] action create (up to date)
2147 * file[/opt/gitlab/sv/alertmanager/down] action nothing (skipped due to action :nothing)
2148 * directory[/opt/gitlab/service] action create (up to date)
2149 * link[/opt/gitlab/service/alertmanager] action create[2023-07-20T03:20:47+00:00] INFO: link[/opt/gitlab/service/alertmanager] created
2150 - create symlink at /opt/gitlab/service/alertmanager to /opt/gitlab/sv/alertmanager
2151 * ruby_block[wait for alertmanager service socket] action run[2023-07-20T03:20:51+00:00] INFO: ruby_block[wait for alertmanager service socket] called
2152 - execute the ruby block wait for alertmanager service socket
2153 * file[/var/log/gitlab/alertmanager/current] action touch (skipped due to only_if)
2154[2023-07-20T03:20:52+00:00] INFO: ruby_block[reload_log_service] called
2155 - execute the ruby block reload_log_service
2156[2023-07-20T03:20:52+00:00] INFO: template[/opt/gitlab/sv/alertmanager/log/run] sending run action to ruby_block[restart_log_service] (delayed)
2157 * ruby_block[restart_log_service] action run
2158 * ruby_block[restart_service] action nothing (skipped due to action :nothing)
2159 * ruby_block[restart_log_service] action nothing (skipped due to action :nothing)
2160 * ruby_block[reload_log_service] action nothing (skipped due to action :nothing)
2161 * directory[/opt/gitlab/sv/alertmanager] action create (up to date)
2162 * template[/opt/gitlab/sv/alertmanager/run] action create (up to date)
2163 * directory[/opt/gitlab/sv/alertmanager/log] action create (up to date)
2164 * directory[/opt/gitlab/sv/alertmanager/log/main] action create (up to date)
2165 * template[/opt/gitlab/sv/alertmanager/log/config] action create (up to date)
2166 * ruby_block[verify_chown_persisted_on_alertmanager] action nothing (skipped due to action :nothing)
2167 * link[/var/log/gitlab/alertmanager/config] action create (up to date)
2168 * template[/opt/gitlab/sv/alertmanager/log/run] action create (up to date)
2169 * directory[/opt/gitlab/sv/alertmanager/env] action create (up to date)
2170 * ruby_block[Delete unmanaged env files for alertmanager service] action run (skipped due to only_if)
2171 * template[/opt/gitlab/sv/alertmanager/check] action create (skipped due to only_if)
2172 * template[/opt/gitlab/sv/alertmanager/finish] action create (skipped due to only_if)
2173 * directory[/opt/gitlab/sv/alertmanager/control] action create (up to date)
2174 * link[/opt/gitlab/init/alertmanager] action create (up to date)
2175 * file[/opt/gitlab/sv/alertmanager/down] action nothing (skipped due to action :nothing)
2176 * directory[/opt/gitlab/service] action create (up to date)
2177 * link[/opt/gitlab/service/alertmanager] action create (up to date)
2178 * ruby_block[wait for alertmanager service socket] action run (skipped due to not_if)
2179 * file[/var/log/gitlab/alertmanager/current] action touch (skipped due to only_if)
2180[2023-07-20T03:20:52+00:00] INFO: ruby_block[restart_log_service] called
2181 - execute the ruby block restart_log_service
2182 * directory[/opt/gitlab/service] action create (up to date)
2183 * link[/opt/gitlab/service/alertmanager] action create (up to date)
2184 * ruby_block[wait for alertmanager service socket] action run (skipped due to not_if)
2185 * file[/var/log/gitlab/alertmanager/current] action touch (skipped due to only_if)
2186
2187 * execute[/opt/gitlab/bin/gitlab-ctl start alertmanager] action run
2188 [execute] ok: run: alertmanager: (pid 757) 3s
2189[2023-07-20T03:20:53+00:00] INFO: execute[/opt/gitlab/bin/gitlab-ctl start alertmanager] ran successfully
2190 - execute /opt/gitlab/bin/gitlab-ctl start alertmanager
2191Recipe: monitoring::postgres-exporter_disable
2192 * service[postgres-exporter] action nothing (skipped due to action :nothing)
2193 * runit_service[postgres-exporter] action disable
2194 * ruby_block[disable postgres-exporter] action run (skipped due to only_if)
2195 (up to date)
2196 * consul_service[postgres-exporter] action delete
2197 * file[/var/opt/gitlab/consul/config.d/postgres-exporter-service.json] action delete (up to date)
2198 (up to date)
2199Recipe: monitoring::grafana_disable
2200 * service[grafana] action nothing (skipped due to action :nothing)
2201 * runit_service[grafana] action disable
2202 * ruby_block[disable grafana] action run (skipped due to only_if)
2203 (up to date)
2204Recipe: gitlab::database_reindexing_disable
2205 * crond_job[database-reindexing] action delete
2206 * file[/var/opt/gitlab/crond/database-reindexing] action delete (up to date)
2207 (up to date)
2208[2023-07-20T03:20:53+00:00] INFO: templatesymlink[Create a gitlab.yml and create a symlink to Rails root] sending run action to execute[clear the gitlab-rails cache] (delayed)
2209Recipe: gitlab::gitlab-rails
2210 * execute[clear the gitlab-rails cache] action run (skipped due to not_if)
2211[2023-07-20T03:20:53+00:00] INFO: version_file[Create version file for Gitaly] sending hup action to runit_service[gitaly] (delayed)
2212Recipe: gitaly::enable
2213 * runit_service[gitaly] action hup[2023-07-20T03:20:53+00:00] INFO: runit_service[gitaly] signalled (HUP)
2214[2023-07-20T03:20:53+00:00] INFO: runit_service[gitaly] sent hup
2215 - send hup to runit_service[gitaly]
2216[2023-07-20T03:20:53+00:00] INFO: version_file[Create version file for Gitlab KAS] sending restart action to runit_service[gitlab-kas] (delayed)
2217Recipe: gitlab-kas::enable
2218 * runit_service[gitlab-kas] action restart
2219
2220 ================================================================================
2221 Error executing action `restart` on resource 'runit_service[gitlab-kas]'
2222 ================================================================================
2223
2224 Mixlib::ShellOut::ShellCommandFailed
2225 ------------------------------------
2226 Expected process to exit with [0], but received '1'
2227 ---- Begin output of /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/gitlab-kas ----
2228 STDOUT: timeout: down: /opt/gitlab/service/gitlab-kas: 0s, normally up, want up
2229 STDERR:
2230 ---- End output of /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/gitlab-kas ----
2231 Ran /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/gitlab-kas returned 1
2232
2233 Cookbook Trace: (most recent call first)
2234 ----------------------------------------
2235 /opt/gitlab/embedded/cookbooks/cache/cookbooks/runit/libraries/helpers.rb:136:in `safe_sv_shellout!'
2236 /opt/gitlab/embedded/cookbooks/cache/cookbooks/runit/libraries/helpers.rb:164:in `restart_service'
2237 /opt/gitlab/embedded/cookbooks/cache/cookbooks/runit/libraries/provider_runit_service.rb:368:in `block in <class:RunitService>'
2238
2239 Resource Declaration:
2240 ---------------------
2241 suppressed sensitive resource output
2242
2243 Compiled Resource:
2244 ------------------
2245 suppressed sensitive resource output
2246
2247 System Info:
2248 ------------
2249 chef_version=17.10.0
2250 platform=ubuntu
2251 platform_version=22.04
2252 ruby=ruby 3.0.6p216 (2023-03-30 revision 23a532679b) [x86_64-linux]
2253 program_name=/opt/gitlab/embedded/bin/cinc-client
2254 executable=/opt/gitlab/embedded/bin/cinc-client
2255
2256[2023-07-20T03:21:24+00:00] INFO: env_dir[/opt/gitlab/etc/prometheus/env] sending restart action to runit_service[prometheus] (delayed)
2257Recipe: monitoring::prometheus
2258 * runit_service[prometheus] action restart (up to date)
2259[2023-07-20T03:21:24+00:00] INFO: file[Prometheus config] sending run action to execute[reload prometheus] (delayed)
2260 * execute[reload prometheus] action run[2023-07-20T03:21:25+00:00] INFO: execute[reload prometheus] ran successfully
2261 - execute /opt/gitlab/bin/gitlab-ctl hup prometheus
2262[2023-07-20T03:21:25+00:00] INFO: env_dir[/opt/gitlab/etc/alertmanager/env] sending restart action to runit_service[alertmanager] (delayed)
2263Recipe: monitoring::alertmanager
2264 * runit_service[alertmanager] action restart (up to date)
2265Running handlers:
2266There was an error running gitlab-ctl reconfigure:
2267runit_service[gitlab-kas] (gitlab-kas::enable line 148) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
2268---- Begin output of /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/gitlab-kas ----
2269STDOUT: timeout: down: /opt/gitlab/service/gitlab-kas: 0s, normally up, want up
2270STDERR:
2271---- End output of /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/gitlab-kas ----
2272Ran /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/gitlab-kas returned 1
2273[2023-07-20T03:21:25+00:00] ERROR: Running exception handlers
2274Running handlers complete
2275[2023-07-20T03:21:25+00:00] ERROR: Exception handlers complete
2276Infra Phase failed. 258 resources updated in 01 minutes 13 seconds
2277[2023-07-20T03:21:25+00:00] FATAL: Stacktrace dumped to /opt/gitlab/embedded/cookbooks/cache/cinc-stacktrace.out
2278[2023-07-20T03:21:25+00:00] FATAL: ---------------------------------------------------------------------------------------
2279[2023-07-20T03:21:25+00:00] FATAL: PLEASE PROVIDE THE CONTENTS OF THE stacktrace.out FILE (above) IF YOU FILE A BUG REPORT
2280[2023-07-20T03:21:25+00:00] FATAL: ---------------------------------------------------------------------------------------
2281[2023-07-20T03:21:25+00:00] FATAL: Mixlib::ShellOut::ShellCommandFailed: runit_service[gitlab-kas] (gitlab-kas::enable line 148) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
2282---- Begin output of /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/gitlab-kas ----
2283STDOUT: timeout: down: /opt/gitlab/service/gitlab-kas: 0s, normally up, want up
2284STDERR:
2285---- End output of /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/gitlab-kas ----
2286Ran /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/gitlab-kas returned 1
2287[Jul 20 2023 03:21:26 UTC (Gitlab QA)] WARN -- Retry instance_no_teardown due to Support::ShellCommand::StatusError -- attempt 1
2288[Jul 20 2023 03:21:26 UTC (Gitlab QA)] INFO -- Shell command: `docker ps -f name=gitaly3`
2289[Jul 20 2023 03:21:26 UTC (Gitlab QA)] INFO -- Shell command: `docker login --username "gitlab-ci-token" --password "*****" registry.gitlab.com`
2290[Jul 20 2023 03:21:27 UTC (Gitlab QA)] INFO -- Shell command: `docker pull -q registry.gitlab.com/gitlab-org/build/omnibus-gitlab-mirror/gitlab-ce:3913740266a59293daf9335e77aacbda3b1a42f1`
2291[Jul 20 2023 03:21:27 UTC (Gitlab QA)] INFO -- Shell command: `docker container list --all --format '{{.Names}}' --filter name=^gitaly3$`
2292[Jul 20 2023 03:21:27 UTC (Gitlab QA)] INFO -- Shell command: `docker rm -f gitaly3`
2293[Jul 20 2023 03:21:27 UTC (Gitlab QA)] INFO -- Shell command: `docker network list --format '{{.Name}}' --filter name=^test$`
2294[Jul 20 2023 03:21:27 UTC (Gitlab QA)] INFO -- Shell command: `docker run -d --name gitaly3 --net test --hostname gitaly3.test -p 8075 --volume /builds/gitlab-org/gitlab/gitlab-qa-run-2023-07-20-03-13-21-3545a9c0/gitaly3/logs:/var/log/gitlab:z registry.gitlab.com/gitlab-org/build/omnibus-gitlab-mirror/gitlab-ce:3913740266a59293daf9335e77aacbda3b1a42f1`
2295[Jul 20 2023 03:21:28 UTC (Gitlab QA)] INFO -- Shell command: `docker exec gitaly3 bash -c "echo \"postgresql['enable'] = false;
2296redis['enable'] = false;
2297nginx['enable'] = false;
2298puma['enable'] = false;
2299sidekiq['enable'] = false;
2300gitlab_workhorse['enable'] = false;
2301gitlab_rails['rake_cache_clear'] = false;
2302gitlab_rails['auto_migrate'] = false;
2303gitlab_exporter['enable'] = false;
2304praefect['enable'] = false;
2305prometheus['enable'] = true;
2306gitaly['enable'] = true;
2307gitaly['configuration'] = {
2308 'listen_addr': '0.0.0.0:8075',
2309 'prometheus_listen_addr': '0.0.0.0:9236',
2310 'auth_token': 'PRAEFECT_INTERNAL_TOKEN'
2311}
2312gitlab_shell['secret_token'] = 'GITLAB_SHELL_SECRET_TOKEN';
2313gitlab_rails['internal_api_url'] = 'http://gitlab-gitaly-cluster.test';
2314git_data_dirs({
2315 'gitaly1' => {
2316 'path' => '/var/opt/gitlab/git-data'
2317 },
2318 'gitaly2' => {
2319 'path' => '/var/opt/gitlab/git-data'
2320 },
2321 'gitaly3' => {
2322 'path' => '/var/opt/gitlab/git-data'
2323 }
2324});
2325\" > /etc/gitlab/gitlab.rb;"`
2326[Jul 20 2023 03:21:28 UTC (Gitlab QA)] INFO -- Shell command: `docker attach --sig-proxy=false gitaly3`
2327[Jul 20 2023 03:22:24 UTC (Gitlab QA)] INFO -- Shell command: `docker exec gitaly3 bash -c "mkdir -p /opt/gitlab/embedded/service/gitlab-shell/hooks/pre-receive.d"`
2328[Jul 20 2023 03:22:24 UTC (Gitlab QA)] INFO -- Shell command: `docker exec gitaly3 bash -c "echo '#!/usr/bin/env bash
2329if [[ \$GL_PROJECT_PATH =~ 'reject-prereceive' ]]; then
2330 echo 'GL-HOOK-ERR: Custom error message rejecting prereceive hook for projects with GL_PROJECT_PATH matching pattern reject-prereceive'
2331 exit 1
2332fi
2333' > /opt/gitlab/embedded/service/gitlab-shell/hooks/pre-receive.d/pre-receive.d;"`
2334[Jul 20 2023 03:22:24 UTC (Gitlab QA)] INFO -- Shell command: `docker exec gitaly3 bash -c "chmod +x /opt/gitlab/embedded/service/gitlab-shell/hooks/pre-receive.d/*"`
2335[Jul 20 2023 03:22:25 UTC (Gitlab QA)] INFO -- Shell command: `docker login --username "gitlab-ci-token" --password "*****" registry.gitlab.com`
2336[Jul 20 2023 03:22:25 UTC (Gitlab QA)] INFO -- Shell command: `docker pull -q registry.gitlab.com/gitlab-org/build/omnibus-gitlab-mirror/gitlab-ce:3913740266a59293daf9335e77aacbda3b1a42f1`
2337[Jul 20 2023 03:22:25 UTC (Gitlab QA)] INFO -- Shell command: `docker container list --all --format '{{.Names}}' --filter name=^praefect$`
2338[Jul 20 2023 03:22:25 UTC (Gitlab QA)] INFO -- Shell command: `docker network list --format '{{.Name}}' --filter name=^test$`
2339[Jul 20 2023 03:22:25 UTC (Gitlab QA)] INFO -- Shell command: `docker run -d --name praefect --net test --hostname praefect.test -p 2305 --volume /builds/gitlab-org/gitlab/gitlab-qa-run-2023-07-20-03-13-21-3545a9c0/praefect/logs:/var/log/gitlab:z registry.gitlab.com/gitlab-org/build/omnibus-gitlab-mirror/gitlab-ce:3913740266a59293daf9335e77aacbda3b1a42f1`
2340[Jul 20 2023 03:22:27 UTC (Gitlab QA)] INFO -- Shell command: `docker exec praefect bash -c "echo \"postgresql['enable'] = false;
2341redis['enable'] = false;
2342nginx['enable'] = false;
2343puma['enable'] = false;
2344sidekiq['enable'] = false;
2345gitlab_workhorse['enable'] = false;
2346gitlab_rails['rake_cache_clear'] = false;
2347gitlab_rails['auto_migrate'] = false;
2348gitlab_exporter['enable'] = false;
2349gitaly['enable'] = false;
2350prometheus['enable'] = true;
2351praefect['enable'] = true;
2352praefect['configuration'] = {
2353 listen_addr: '0.0.0.0:2305',
2354 prometheus_listen_addr: '0.0.0.0:9652',
2355 auth: {
2356 token: 'PRAEFECT_EXTERNAL_TOKEN'
2357 },
2358 reconciliation: {
2359 scheduling_interval: '10s'
2360 },
2361 database: {
2362 host: 'postgres.test',
2363 port: 5432,
2364 user: 'postgres',
2365 password: 'SQL_PASSWORD',
2366 dbname: 'praefect_production',
2367 sslmode: 'disable'
2368 },
2369 virtual_storage: [
2370 {
2371 name: 'default',
2372 node: [
2373 {
2374 'storage': 'gitaly1',
2375 'address': 'tcp://gitaly1.test:8075',
2376 'token': 'PRAEFECT_INTERNAL_TOKEN'
2377 },
2378 {
2379 'storage': 'gitaly2',
2380 'address': 'tcp://gitaly2.test:8075',
2381 'token': 'PRAEFECT_INTERNAL_TOKEN'
2382 },
2383 {
2384 'storage': 'gitaly3',
2385 'address': 'tcp://gitaly3.test:8075',
2386 'token': 'PRAEFECT_INTERNAL_TOKEN'
2387 }
2388 ],
2389 }
2390 ]
2391}
2392\" > /etc/gitlab/gitlab.rb;"`
2393[Jul 20 2023 03:22:27 UTC (Gitlab QA)] INFO -- Shell command: `docker attach --sig-proxy=false praefect`
2394[Jul 20 2023 03:23:46 UTC (Gitlab QA)] ERROR -- Shell command output:
2395Cleaning stale PIDs & sockets
2396cat: /var/opt/gitlab/gitlab-rails/VERSION: No such file or directory
2397Generating ssh_host_rsa_key...
2398Generating public/private rsa key pair.
2399Your identification has been saved in /etc/gitlab/ssh_host_rsa_key
2400Your public key has been saved in /etc/gitlab/ssh_host_rsa_key.pub
2401The key fingerprint is:
2402SHA256:XIc3ROIldZVAx5qASnsWRynWgYuAmtHI+XqgCfcWiAA root@praefect.test
2403The key's randomart image is:
2404+---[RSA 3072]----+
2405|E + . B=Oo+oo|
2406|.= o . . B @ ..o |
2407|..=. o = O = o |
2408|o+o.. = = o + |
2409|ooo. . S |
2410|o. .o |
2411| .. |
2412| |
2413| |
2414+----[SHA256]-----+
2415Generating ssh_host_ecdsa_key...
2416Generating public/private ecdsa key pair.
2417Your identification has been saved in /etc/gitlab/ssh_host_ecdsa_key
2418Your public key has been saved in /etc/gitlab/ssh_host_ecdsa_key.pub
2419The key fingerprint is:
2420SHA256:LMKvEsuW+ortlsGMKPutY06sMbhljzXbkQ2RC/O7oJc root@praefect.test
2421The key's randomart image is:
2422+---[ECDSA 256]---+
2423| |
2424| . |
2425| o o |
2426| . + + |
2427|.+ o = S |
2428|=.= o * |
2429|=o=*+.= . |
2430|oX@*E* o |
2431|*@X== o |
2432+----[SHA256]-----+
2433Generating ssh_host_ed25519_key...
2434Generating public/private ed25519 key pair.
2435Your identification has been saved in /etc/gitlab/ssh_host_ed25519_key
2436Your public key has been saved in /etc/gitlab/ssh_host_ed25519_key.pub
2437The key fingerprint is:
2438SHA256:uvCOiERo7UTZ5KV2Qh1QiISYvCD4T69YlYJPg/eEYeY root@praefect.test
2439The key's randomart image is:
2440+--[ED25519 256]--+
2441|+.o..+=+. |
2442|*o .X.o. |
2443|o..O O o |
2444|..* E * |
2445|.o O B S |
2446|o o + o. |
2447| . +... |
2448|.....+ . |
2449|. . ..+ |
2450+----[SHA256]-----+
2451Preparing services...
2452Starting services...
2453Configuring GitLab...
2454/opt/gitlab/embedded/bin/runsvdir-start: line 37: /proc/sys/fs/file-max: Read-only file system
2455[2023-07-20T03:22:33+00:00] INFO: Started Cinc Zero at chefzero://localhost:1 with repository at /opt/gitlab/embedded (One version per cookbook)
2456Cinc Client, version 17.10.0
2457Patents: https://www.chef.io/patents
2458Infra Phase starting
2459[2023-07-20T03:22:33+00:00] INFO: *** Cinc Client 17.10.0 ***
2460[2023-07-20T03:22:33+00:00] INFO: Platform: x86_64-linux
2461[2023-07-20T03:22:33+00:00] INFO: Cinc-client pid: 37
2462[2023-07-20T03:22:33+00:00] INFO: Setting the run_list to ["recipe[gitlab]"] from CLI options
2463[2023-07-20T03:22:33+00:00] INFO: Run List is [recipe[gitlab]]
2464[2023-07-20T03:22:33+00:00] INFO: Run List expands to [gitlab]
2465[2023-07-20T03:22:33+00:00] INFO: Starting Cinc Client Run for praefect.test
2466[2023-07-20T03:22:33+00:00] INFO: Running start handlers
2467[2023-07-20T03:22:33+00:00] INFO: Start handlers complete.
2468Resolving cookbooks for run list: ["gitlab"]
2469[2023-07-20T03:22:34+00:00] INFO: Loading cookbooks [gitlab@0.0.1, package@0.1.0, logrotate@0.1.0, postgresql@0.1.0, redis@0.1.0, monitoring@0.1.0, registry@0.1.0, mattermost@0.1.0, consul@0.1.0, gitaly@0.1.0, praefect@0.1.0, gitlab-kas@0.1.0, gitlab-pages@0.1.0, letsencrypt@0.1.0, nginx@0.1.0, runit@5.1.7, acme@4.1.6, crond@0.1.0]
2470Synchronizing cookbooks:
2471[2023-07-20T03:22:34+00:00] INFO: Storing updated cookbooks/gitlab/recipes/add_trusted_certs.rb in the cache.
2472[2023-07-20T03:22:34+00:00] INFO: Storing updated cookbooks/gitlab/recipes/database_migrations.rb in the cache.
2473[2023-07-20T03:22:34+00:00] INFO: Storing updated cookbooks/gitlab/recipes/default.rb in the cache.
2474[2023-07-20T03:22:34+00:00] INFO: Storing updated cookbooks/gitlab/recipes/config.rb in the cache.
2475[2023-07-20T03:22:34+00:00] INFO: Storing updated cookbooks/gitlab/recipes/database_reindexing_enable.rb in the cache.
2476[2023-07-20T03:22:34+00:00] INFO: Storing updated cookbooks/gitlab/recipes/gitlab-healthcheck.rb in the cache.
2477[2023-07-20T03:22:34+00:00] INFO: Storing updated cookbooks/gitlab/recipes/bootstrap.rb in the cache.
2478[2023-07-20T03:22:34+00:00] INFO: Storing updated cookbooks/gitlab/recipes/database_reindexing_disable.rb in the cache.
2479[2023-07-20T03:22:34+00:00] INFO: Storing updated cookbooks/gitlab/recipes/gitlab-rails.rb in the cache.
2480[2023-07-20T03:22:34+00:00] INFO: Storing updated cookbooks/gitlab/recipes/bootstrap_disable.rb in the cache.
2481[2023-07-20T03:22:34+00:00] INFO: Storing updated cookbooks/gitlab/recipes/gitlab-shell.rb in the cache.
2482[2023-07-20T03:22:34+00:00] INFO: Storing updated cookbooks/gitlab/recipes/gitlab-workhorse.rb in the cache.
2483[2023-07-20T03:22:34+00:00] INFO: Storing updated cookbooks/gitlab/recipes/mailroom_disable.rb in the cache.
2484[2023-07-20T03:22:34+00:00] INFO: Storing updated cookbooks/gitlab/recipes/mailroom.rb in the cache.
2485[2023-07-20T03:22:34+00:00] INFO: Storing updated cookbooks/gitlab/recipes/nginx.rb in the cache.
2486[2023-07-20T03:22:34+00:00] INFO: Storing updated cookbooks/gitlab/recipes/nginx_disable.rb in the cache.
2487[2023-07-20T03:22:34+00:00] INFO: Storing updated cookbooks/gitlab/recipes/letsencrypt_renew.rb in the cache.
2488[2023-07-20T03:22:34+00:00] INFO: Storing updated cookbooks/gitlab/recipes/gitlab-workhorse_disable.rb in the cache.
2489[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab/recipes/puma.rb in the cache.
2490[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab/recipes/puma_disable.rb in the cache.
2491[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab/recipes/remote-syslog.rb in the cache.
2492[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab/recipes/rails_pages_shared_path.rb in the cache.
2493[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab/recipes/remote-syslog_disable.rb in the cache.
2494[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab/recipes/remove_accounts.rb in the cache.
2495[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab/recipes/show_config.rb in the cache.
2496[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab/recipes/storage-check.rb in the cache.
2497[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab/recipes/sidekiq.rb in the cache.
2498[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab/templates/default/cable.yml.erb in the cache.
2499[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab/templates/default/database.yml.erb in the cache.
2500[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab/templates/default/gitconfig-system.erb in the cache.
2501[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab/recipes/sidekiq_disable.rb in the cache.
2502[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab/templates/default/gitlab-healthcheck-rc.erb in the cache.
2503[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab/templates/default/gitlab-rails-error.html.erb in the cache.
2504[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab/templates/default/gitlab-rails-rc.erb in the cache.
2505[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab/recipes/web-server.rb in the cache.
2506[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab/recipes/storage-check_disable.rb in the cache.
2507[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab/recipes/selinux.rb in the cache.
2508[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab/recipes/users.rb in the cache.
2509[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab/templates/default/gitconfig.erb in the cache.
2510[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab/templates/default/gitlab-shell-config.yml.erb in the cache.
2511[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab/templates/default/make_metrics_rundir.erb in the cache.
2512[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab/templates/default/gitlab.yml.erb in the cache.
2513[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab/templates/default/mount_point_check.erb in the cache.
2514[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab/templates/default/nginx-gitlab-kas-http.conf.erb in the cache.
2515[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab/templates/default/nginx-gitlab-http.conf.erb in the cache.
2516[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab/templates/default/nginx-gitlab-registry-http.conf.erb in the cache.
2517[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab/templates/default/nginx-gitlab-health.conf.erb in the cache.
2518[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab/templates/default/nginx-gitlab-mattermost-http.conf.erb in the cache.
2519[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab/templates/default/redis.yml.erb in the cache.
2520[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab/templates/default/remote_syslog.yml.erb in the cache.
2521[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab/templates/default/resque.yml.erb in the cache.
2522[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab/templates/default/secret_token.erb in the cache.
2523[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab/templates/default/secrets.yml.erb in the cache.
2524[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab/templates/default/nginx-status.conf.erb in the cache.
2525[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab/templates/default/nginx-gitlab-pages-http.conf.erb in the cache.
2526[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab/templates/default/nginx.conf.erb in the cache.
2527[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab/templates/default/sv-gitlab-sshd-log-run.erb in the cache.
2528[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab/templates/default/sv-gitlab-sshd-run.erb in the cache.
2529[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab/templates/default/puma.rb.erb in the cache.
2530[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab/templates/default/sv-gitlab-sshd-log-config.erb in the cache.
2531[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab/templates/default/sv-gitlab-workhorse-log-run.erb in the cache.
2532[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab/templates/default/sv-gitlab-workhorse-run.erb in the cache.
2533[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab/templates/default/sv-mailroom-finish.erb in the cache.
2534[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab/templates/default/sv-mailroom-log-config.erb in the cache.
2535[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab/templates/default/smtp_settings.rb.erb in the cache.
2536[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab/templates/default/sv-gitlab-workhorse-log-config.erb in the cache.
2537[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab/templates/default/sv-mailroom-run.erb in the cache.
2538[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab/templates/default/sv-mailroom-log-run.erb in the cache.
2539[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab/templates/default/sv-puma-h.erb in the cache.
2540[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab/templates/default/sv-puma-log-run.erb in the cache.
2541[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab/templates/default/sv-puma-run.erb in the cache.
2542[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab/templates/default/sv-puma-t.erb in the cache.
2543[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab/templates/default/sv-puma-log-config.erb in the cache.
2544[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab/templates/default/sv-sidekiq-log-config.erb in the cache.
2545[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab/templates/default/sv-remote-syslog-run.erb in the cache.
2546[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab/templates/default/sv-remote-syslog-log-config.erb in the cache.
2547[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab/templates/default/sv-sidekiq-log-run.erb in the cache.
2548[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab/templates/default/sv-storage-check-log-run.erb in the cache.
2549[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab/templates/default/sv-sidekiq-run.erb in the cache.
2550[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab/templates/default/sv-remote-syslog-log-run.erb in the cache.
2551[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab/templates/default/sv-storage-check-log-config.erb in the cache.
2552[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab/templates/default/workhorse-config.toml.erb in the cache.
2553[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab/templates/default/sv-storage-check-run.erb in the cache.
2554[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab/README.md in the cache.
2555[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab/resources/database_objects.rb in the cache.
2556[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab/resources/rails_migration.rb in the cache.
2557[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab/resources/puma_config.rb in the cache.
2558[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab/libraries/bash_hide_env.rb in the cache.
2559[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab/resources/sidekiq_service.rb in the cache.
2560[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab/libraries/account_helper.rb in the cache.
2561[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab/libraries/gitlab_exporter.rb in the cache.
2562[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab/libraries/gitlab_mattermost.rb in the cache.
2563[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab/libraries/gitlab_shell.rb in the cache.
2564[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab/libraries/gitlab_rails.rb in the cache.
2565[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab/libraries/gitlab_sshd_helper.rb in the cache.
2566[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab/libraries/gitlab_workhorse.rb in the cache.
2567[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab/libraries/helper.rb in the cache.
2568[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab/libraries/helpers/authorizer_helper.rb in the cache.
2569[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab/libraries/helpers/base_helper.rb in the cache.
2570[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab/libraries/helpers/geo_pg_helper.rb in the cache.
2571[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab/libraries/helpers/gitlab_rails.rb in the cache.
2572[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab/libraries/helpers/base_pg_helper.rb in the cache.
2573[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab/libraries/helpers/gitlab_rails_env_helper.rb in the cache.
2574[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab/libraries/helpers/grafana_helper.rb in the cache.
2575[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab/libraries/helpers/metrics_exporter_helper.rb in the cache.
2576[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab/libraries/helpers/gitlab_workhorse_helper.rb in the cache.
2577[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab/libraries/helpers/pg_status_helper.rb in the cache.
2578[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab/libraries/helpers/pg_helper.rb in the cache.
2579[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab/libraries/helpers/web_server_helper.rb in the cache.
2580[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab/libraries/incoming_email.rb in the cache.
2581[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab/libraries/patroni.rb in the cache.
2582[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab/libraries/logging.rb in the cache.
2583[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab/libraries/nginx.rb in the cache.
2584[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab/libraries/logfiles_helper.rb in the cache.
2585[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab/libraries/pg_version.rb in the cache.
2586[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab/libraries/mailroom_helper.rb in the cache.
2587[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab/libraries/puma.rb in the cache.
2588[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab/libraries/postgresql.rb in the cache.
2589[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab/libraries/redis.rb in the cache.
2590[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab/libraries/rails_migration_helper.rb in the cache.
2591[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab/libraries/redis_helper.rb in the cache.
2592[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab/libraries/redis_uri.rb in the cache.
2593[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab/libraries/sidekiq.rb in the cache.
2594[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab/libraries/registry.rb in the cache.
2595[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab/libraries/storage_check.rb in the cache.
2596[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab/libraries/smtp_helper.rb in the cache.
2597[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab/attributes/default.rb in the cache.
2598[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab/metadata.rb in the cache.
2599[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/package/recipes/runit_systemd.rb in the cache.
2600[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/package/recipes/runit.rb in the cache.
2601[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/package/templates/default/gitlab-runsvdir.service.erb in the cache.
2602[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/package/resources/account.rb in the cache.
2603[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/package/recipes/runit_upstart.rb in the cache.
2604[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/package/recipes/sysctl.rb in the cache.
2605[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/package/resources/gitlab_sysctl.rb in the cache.
2606[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/package/recipes/runit_sysvinit.rb in the cache.
2607 [2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/package/resources/env_dir.rb in the cache.
2608- gitlab (0.0.1)
2609[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/package/resources/storage_directory.rb in the cache.
2610[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/package/resources/templatesymlink.rb in the cache.
2611[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/package/libraries/certificate_helper.rb in the cache.
2612[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/package/libraries/config/gitlab.rb in the cache.
2613[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/package/libraries/config/roles/geo_primary.rb in the cache.
2614[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/package/resources/version_file.rb in the cache.
2615[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/package/libraries/config/roles/default.rb in the cache.
2616[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/package/libraries/config/roles/consul.rb in the cache.
2617[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/package/libraries/config/roles/geo_secondary.rb in the cache.
2618[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/package/libraries/config/roles/application.rb in the cache.
2619[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/package/libraries/config/roles/pages.rb in the cache.
2620[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/package/libraries/config/roles/monitoring.rb in the cache.
2621[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/package/libraries/config/roles/postgres.rb in the cache.
2622[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/package/libraries/config/roles/pgbouncer.rb in the cache.
2623[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/package/libraries/config/roles/redis_master.rb in the cache.
2624[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/package/libraries/config/roles/patroni.rb in the cache.
2625[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/package/libraries/config/roles/sidekiq.rb in the cache.
2626[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/package/libraries/config/roles/redis_sentinel.rb in the cache.
2627[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/package/libraries/config/roles/spamcheck.rb in the cache.
2628[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/package/libraries/config_mash.rb in the cache.
2629[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/package/libraries/config/services.rb in the cache.
2630[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/package/libraries/formatters/gitlab.rb in the cache.
2631[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/package/libraries/deprecations.rb in the cache.
2632[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/package/libraries/gitlab_cluster.rb in the cache.
2633[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/package/libraries/handlers/gitlab.rb in the cache.
2634[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/package/libraries/helper.rb in the cache.
2635[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/package/libraries/helpers/logging_helper.rb in the cache.
2636[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/package/libraries/helpers/output_helper.rb in the cache.
2637[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/package/libraries/helpers/node_helper.rb in the cache.
2638[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/package/libraries/helpers/roles_helper.rb in the cache.
2639[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/package/libraries/helpers/secrets_helper.rb in the cache.
2640[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/package/libraries/helpers/selinux_distro_helper.rb in the cache.
2641[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/package/libraries/helpers/services_helper.rb in the cache.
2642[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/package/libraries/helpers/version_helper.rb in the cache.
2643[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/package/libraries/helpers/selinux_helper.rb in the cache.
2644[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/package/libraries/helpers/systemd_helper.rb in the cache.
2645[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/package/libraries/helpers/shell_out_helper.rb in the cache.
2646[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/package/libraries/object_proxy.rb in the cache.
2647[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/package/libraries/omnibus_helper.rb in the cache.
2648[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/package/libraries/storage_directory_helper.rb in the cache.
2649[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/package/files/default/gitlab-runsvdir.conf in the cache.
2650[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/package/libraries/package.rb in the cache.
2651[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/package/metadata.rb in the cache.
2652[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/package/attributes/default.rb in the cache.
2653[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/logrotate/recipes/disable.rb in the cache.
2654[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/logrotate/recipes/folders_and_configs.rb in the cache.
2655[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/package/libraries/settings_dsl.rb in the cache.
2656[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/logrotate/templates/default/logrotate-service.erb in the cache.
2657[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/logrotate/recipes/enable.rb in the cache.
2658[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/logrotate/templates/default/sv-logrotate-log-config.erb in the cache.
2659[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/logrotate/templates/default/logrotate.conf.erb in the cache.
2660[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/logrotate/templates/default/sv-logrotate-log-run.erb in the cache.
2661[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/logrotate/attributes/default.rb in the cache.
2662[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/logrotate/templates/default/sv-logrotate-t.erb in the cache.
2663[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/logrotate/templates/default/sv-logrotate-run.erb in the cache.
2664 - package (0.1.0)
2665[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/postgresql/recipes/bin.rb in the cache.
2666[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/logrotate/metadata.rb in the cache.
2667[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/postgresql/recipes/disable.rb in the cache.
2668[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/postgresql/recipes/directory_locations.rb in the cache.
2669[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/postgresql/recipes/enable.rb in the cache.
2670 - logrotate (0.1.0)
2671[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/postgresql/recipes/standalone.rb in the cache.
2672[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/postgresql/recipes/user.rb in the cache.
2673[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/postgresql/templates/default/pg_ident.conf.erb in the cache.
2674[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/postgresql/templates/default/pg_hba.conf.erb in the cache.
2675[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/postgresql/recipes/sysctl.rb in the cache.
2676[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/postgresql/templates/default/gitlab-psql-rc.erb in the cache.
2677[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/postgresql/templates/default/postgresql.conf.erb in the cache.
2678[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/postgresql/templates/default/postgresql-runtime.conf.erb in the cache.
2679[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/postgresql/templates/default/sv-postgresql-t.erb in the cache.
2680[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/postgresql/templates/default/sv-postgresql-run.erb in the cache.
2681[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/postgresql/README.md in the cache.
2682[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/postgresql/templates/default/sv-postgresql-log-run.erb in the cache.
2683[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/postgresql/templates/default/sv-postgresql-log-config.erb in the cache.
2684[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/postgresql/resources/extension.rb in the cache.
2685[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/postgresql/resources/database.rb in the cache.
2686[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/postgresql/resources/config.rb in the cache.
2687[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/postgresql/resources/query.rb in the cache.
2688[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/postgresql/resources/schema.rb in the cache.
2689[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/postgresql/resources/user.rb in the cache.
2690[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/postgresql/attributes/default.rb in the cache.
2691[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/redis/recipes/disable.rb in the cache.
2692[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/postgresql/metadata.rb in the cache.
2693[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/redis/recipes/enable.rb in the cache.
2694[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/redis/templates/default/gitlab-redis-cli-rc.erb in the cache.
2695[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/redis/templates/default/redis.conf.erb in the cache.
2696[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/redis/templates/default/sv-redis-log-config.erb in the cache.
2697[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/redis/templates/default/sv-redis-log-run.erb in the cache.
2698 - postgresql (0.1.0)[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/redis/templates/default/sv-redis-run.erb in the cache.
2699[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/redis/attributes/attributes.rb in the cache.
2700[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/redis/resources/service.rb in the cache.
2701[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/redis/metadata.rb in the cache.
2702[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/monitoring/recipes/alertmanager_disable.rb in the cache.
2703[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/monitoring/recipes/alertmanager.rb in the cache.
2704[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/monitoring/recipes/default.rb in the cache.
2705[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/monitoring/recipes/gitlab-exporter_disable.rb in the cache.
2706[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/monitoring/recipes/gitlab-exporter.rb in the cache.
2707[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/monitoring/recipes/grafana_disable.rb in the cache.
2708[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/monitoring/recipes/grafana.rb in the cache.
2709 [2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/monitoring/recipes/node-exporter.rb in the cache.
2710- redis (0.1.0)
2711[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/monitoring/recipes/node-exporter_disable.rb in the cache.
2712[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/monitoring/recipes/pgbouncer-exporter.rb in the cache.
2713[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/monitoring/recipes/postgres-exporter_disable.rb in the cache.
2714[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/monitoring/recipes/pgbouncer-exporter_disable.rb in the cache.
2715[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/monitoring/recipes/postgres-exporter.rb in the cache.
2716[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/monitoring/recipes/redis-exporter.rb in the cache.
2717[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/monitoring/recipes/prometheus_disable.rb in the cache.
2718[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/monitoring/recipes/redis-exporter_disable.rb in the cache.
2719[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/monitoring/recipes/prometheus.rb in the cache.
2720[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/monitoring/templates/gitlab-exporter.yml.erb in the cache.
2721[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/monitoring/templates/grafana_ini.erb in the cache.
2722[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/monitoring/recipes/user.rb in the cache.
2723[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/monitoring/templates/mount_point_check.erb in the cache.
2724[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/monitoring/templates/postgres-queries.yaml.erb in the cache.
2725[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/monitoring/templates/sv-alertmanager-run.erb in the cache.
2726[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/monitoring/templates/rules/node.rules in the cache.
2727[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/monitoring/templates/rules/gitlab.rules in the cache.
2728[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/monitoring/templates/sv-alertmanager-log-run.erb in the cache.
2729[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/monitoring/templates/sv-alertmanager-log-config.erb in the cache.
2730[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/monitoring/templates/sv-gitlab-exporter-log-run.erb in the cache.
2731[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/monitoring/templates/sv-gitlab-exporter-log-config.erb in the cache.
2732[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/monitoring/templates/sv-gitlab-exporter-run.erb in the cache.
2733[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/monitoring/templates/sv-node-exporter-log-run.erb in the cache.
2734[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/monitoring/templates/sv-node-exporter-log-config.erb in the cache.
2735[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/monitoring/templates/sv-node-exporter-run.erb in the cache.
2736[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/monitoring/templates/sv-grafana-log-config.erb in the cache.
2737[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/monitoring/templates/sv-grafana-run.erb in the cache.
2738[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/monitoring/templates/sv-grafana-log-run.erb in the cache.
2739[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/monitoring/templates/sv-pgbouncer-exporter-log-config.erb in the cache.
2740[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/monitoring/templates/sv-pgbouncer-exporter-log-run.erb in the cache.
2741[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/monitoring/templates/sv-pgbouncer-exporter-run.erb in the cache.
2742[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/monitoring/templates/sv-postgres-exporter-log-run.erb in the cache.
2743[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/monitoring/templates/sv-postgres-exporter-log-config.erb in the cache.
2744[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/monitoring/templates/sv-prometheus-log-config.erb in the cache.
2745[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/monitoring/templates/sv-postgres-exporter-run.erb in the cache.
2746[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/monitoring/templates/sv-prometheus-log-run.erb in the cache.
2747[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/monitoring/templates/sv-redis-exporter-log-run.erb in the cache.
2748[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/monitoring/templates/sv-prometheus-run.erb in the cache.
2749[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/monitoring/templates/sv-redis-exporter-log-config.erb in the cache.
2750[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/monitoring/templates/sv-redis-exporter-run.erb in the cache.
2751[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/monitoring/libraries/prometheus.rb in the cache.
2752[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/monitoring/libraries/grafana.rb in the cache.
2753[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/registry/recipes/disable.rb in the cache.
2754[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/monitoring/attributes/default.rb in the cache.
2755[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/registry/recipes/enable.rb in the cache.
2756[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/monitoring/libraries/prometheus_helper.rb in the cache.
2757[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/monitoring/metadata.rb in the cache.
2758[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/registry/templates/default/sv-registry-log-run.erb in the cache.
2759[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/registry/templates/default/registry-config.yml.erb in the cache.
2760[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/registry/templates/default/sv-registry-log-config.erb in the cache.
2761[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/registry/templates/default/sv-registry-run.erb in the cache.
2762[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/registry/attributes/default.rb in the cache.
2763 - monitoring (0.1.0)[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/registry/metadata.rb in the cache.
2764[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/mattermost/recipes/disable.rb in the cache.
2765[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/mattermost/recipes/enable.rb in the cache.
2766[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/mattermost/templates/default/sv-mattermost-log-config.erb in the cache.
2767[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/mattermost/templates/default/sv-mattermost-log-run.erb in the cache.
2768[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/mattermost/templates/default/sv-mattermost-run.erb in the cache.
2769[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/mattermost/libraries/mattermost_helper.rb in the cache.
2770[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/mattermost/metadata.rb in the cache.
2771 - registry (0.1.0)
2772[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/mattermost/attributes/default.rb in the cache.
2773[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/consul/recipes/configure_services.rb in the cache.
2774[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/consul/recipes/disable_service_postgresql.rb in the cache.
2775[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/consul/recipes/disable_daemon.rb in the cache.
2776[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/consul/recipes/enable.rb in the cache.
2777 - mattermost (0.1.0)
2778[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/consul/recipes/enable_daemon.rb in the cache.
2779[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/consul/recipes/enable_service_postgresql.rb in the cache.
2780[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/consul/recipes/disable.rb in the cache.
2781[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/consul/templates/default/sv-consul-log-config.erb in the cache.
2782[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/consul/recipes/watchers.rb in the cache.
2783[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/consul/templates/default/sv-consul-log-run.erb in the cache.
2784[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/consul/templates/default/watcher_scripts/failover_pgbouncer.erb in the cache.
2785[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/consul/templates/default/sv-consul-run.erb in the cache.
2786[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/consul/resources/consul_service.rb in the cache.
2787[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/consul/libraries/watch_helper.rb in the cache.
2788[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/consul/libraries/consul.rb in the cache.
2789[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/consul/libraries/consul_helper.rb in the cache.
2790[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/consul/libraries/failover_helper.rb in the cache.
2791[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/consul/attributes/default.rb in the cache.
2792[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/consul/attributes/watchers.rb in the cache.
2793[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/consul/attributes/services.rb in the cache.
2794[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitaly/templates/default/sv-gitaly-log-run.erb in the cache.
2795[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitaly/templates/default/gitaly-config.toml.erb in the cache.
2796[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/consul/metadata.rb in the cache.
2797[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitaly/recipes/disable.rb in the cache.
2798[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitaly/recipes/enable.rb in the cache.
2799[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitaly/attributes/default.rb in the cache.
2800[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitaly/templates/default/sv-gitaly-log-config.erb in the cache.
2801[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitaly/templates/default/sv-gitaly-run.erb in the cache.
2802[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitaly/metadata.rb in the cache.
2803[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitaly/libraries/gitaly.rb in the cache.
2804[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/praefect/recipes/database_migrations.rb in the cache.
2805[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/praefect/recipes/disable.rb in the cache.
2806 - consul (0.1.0)
2807[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/praefect/recipes/enable.rb in the cache.
2808[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/praefect/attributes/default.rb in the cache.
2809[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/praefect/templates/default/sv-praefect-log-config.erb in the cache.
2810[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/praefect/templates/default/sv-praefect-log-run.erb in the cache.
2811 - gitaly (0.1.0)
2812[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/praefect/templates/default/sv-praefect-run.erb in the cache.
2813[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/praefect/templates/default/praefect-config.toml.erb in the cache.
2814[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/praefect/metadata.rb in the cache.
2815[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab-kas/templates/default/gitlab-kas-config.yml.erb in the cache.
2816[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab-kas/recipes/enable.rb in the cache.
2817[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab-kas/recipes/disable.rb in the cache.
2818[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab-kas/templates/default/sv-gitlab-kas-log-config.erb in the cache.
2819[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab-kas/libraries/gitlab_kas.rb in the cache.
2820 - praefect (0.1.0)
2821[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab-kas/templates/default/sv-gitlab-kas-log-run.erb in the cache.
2822[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab-kas/templates/default/sv-gitlab-kas-run.erb in the cache.
2823[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab-kas/metadata.rb in the cache.
2824[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab-pages/recipes/enable.rb in the cache.
2825[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab-pages/templates/default/gitlab-pages-config.erb in the cache.
2826[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab-pages/templates/default/mount_point_check.erb in the cache.
2827[2023-07-20T03:22:35+00:00] INFO: Storing updated cookbooks/gitlab-kas/attributes/default.rb in the cache.
2828[2023-07-20T03:22:36+00:00] INFO: Storing updated cookbooks/gitlab-pages/recipes/disable.rb in the cache.
2829[2023-07-20T03:22:36+00:00] INFO: Storing updated cookbooks/gitlab-pages/templates/default/sv-gitlab-pages-log-config.erb in the cache.
2830[2023-07-20T03:22:36+00:00] INFO: Storing updated cookbooks/gitlab-pages/templates/default/secret_token.erb in the cache.
2831[2023-07-20T03:22:36+00:00] INFO: Storing updated cookbooks/gitlab-pages/templates/default/sv-gitlab-pages-log-run.erb in the cache.
2832[2023-07-20T03:22:36+00:00] INFO: Storing updated cookbooks/gitlab-pages/libraries/gitlab_pages.rb in the cache.
2833[2023-07-20T03:22:36+00:00] INFO: Storing updated cookbooks/gitlab-pages/templates/default/sv-gitlab-pages-run.erb in the cache.
2834 - gitlab-kas (0.1.0)[2023-07-20T03:22:36+00:00] INFO: Storing updated cookbooks/letsencrypt/recipes/enable.rb in the cache.
2835[2023-07-20T03:22:36+00:00] INFO: Storing updated cookbooks/gitlab-pages/metadata.rb in the cache.
2836[2023-07-20T03:22:36+00:00] INFO: Storing updated cookbooks/gitlab-pages/attributes/default.rb in the cache.
2837[2023-07-20T03:22:36+00:00] INFO: Storing updated cookbooks/letsencrypt/recipes/disable.rb in the cache.
2838[2023-07-20T03:22:36+00:00] INFO: Storing updated cookbooks/letsencrypt/resources/certificate.rb in the cache.
2839[2023-07-20T03:22:36+00:00] INFO: Storing updated cookbooks/letsencrypt/recipes/http_authorization.rb in the cache.
2840[2023-07-20T03:22:36+00:00] INFO: Storing updated cookbooks/letsencrypt/README.md in the cache.
2841[2023-07-20T03:22:36+00:00] INFO: Storing updated cookbooks/letsencrypt/libraries/lets_encrypt.rb in the cache.
2842[2023-07-20T03:22:36+00:00] INFO: Storing updated cookbooks/letsencrypt/libraries/helper.rb in the cache.
2843[2023-07-20T03:22:36+00:00] INFO: Storing updated cookbooks/letsencrypt/attributes/default.rb in the cache.
2844 - gitlab-pages (0.1.0)
2845[2023-07-20T03:22:36+00:00] INFO: Storing updated cookbooks/letsencrypt/metadata.rb in the cache.
2846[2023-07-20T03:22:36+00:00] INFO: Storing updated cookbooks/nginx/recipes/enable.rb in the cache.
2847[2023-07-20T03:22:36+00:00] INFO: Storing updated cookbooks/nginx/templates/default/sv-nginx-log-config.erb in the cache.
2848[2023-07-20T03:22:36+00:00] INFO: Storing updated cookbooks/nginx/templates/default/sv-nginx-log-run.erb in the cache.
2849[2023-07-20T03:22:36+00:00] INFO: Storing updated cookbooks/nginx/templates/default/sv-nginx-run.erb in the cache.
2850[2023-07-20T03:22:36+00:00] INFO: Storing updated cookbooks/nginx/metadata.rb in the cache.
2851[2023-07-20T03:22:36+00:00] INFO: Storing updated cookbooks/runit/README.md in the cache.
2852 - letsencrypt (0.1.0)[2023-07-20T03:22:36+00:00] INFO: Storing updated cookbooks/runit/libraries/resource_runit_service.rb in the cache.
2853[2023-07-20T03:22:36+00:00] INFO: Storing updated cookbooks/runit/libraries/helpers.rb in the cache.
2854[2023-07-20T03:22:36+00:00] INFO: Storing updated cookbooks/runit/libraries/provider_runit_service.rb in the cache.
2855 - nginx (0.1.0)
2856[2023-07-20T03:22:36+00:00] INFO: Storing updated cookbooks/runit/metadata.rb in the cache.
2857[2023-07-20T03:22:36+00:00] INFO: Storing updated cookbooks/acme/templates/acme-challange.nginx.erb in the cache.
2858[2023-07-20T03:22:36+00:00] INFO: Storing updated cookbooks/acme/kitchen.dokken.yml in the cache.
2859[2023-07-20T03:22:36+00:00] INFO: Storing updated cookbooks/acme/recipes/default.rb in the cache.
2860[2023-07-20T03:22:36+00:00] INFO: Storing updated cookbooks/acme/kitchen.digitalocean.yml in the cache.
2861[2023-07-20T03:22:36+00:00] INFO: Storing updated cookbooks/runit/LICENSE in the cache.
2862[2023-07-20T03:22:36+00:00] INFO: Storing updated cookbooks/acme/README.md in the cache.
2863[2023-07-20T03:22:36+00:00] INFO: Storing updated cookbooks/acme/resources/ssl_key.rb in the cache.
2864[2023-07-20T03:22:36+00:00] INFO: Storing updated cookbooks/acme/resources/certificate.rb in the cache.
2865[2023-07-20T03:22:36+00:00] INFO: Storing updated cookbooks/acme/resources/selfsigned.rb in the cache.
2866[2023-07-20T03:22:36+00:00] INFO: Storing updated cookbooks/acme/chefignore in the cache.
2867 [2023-07-20T03:22:36+00:00] INFO: Storing updated cookbooks/acme/kitchen.yml in the cache.
2868[2023-07-20T03:22:36+00:00] INFO: Storing updated cookbooks/acme/attributes/default.rb in the cache.
2869[2023-07-20T03:22:36+00:00] INFO: Storing updated cookbooks/acme/libraries/acme.rb in the cache.
2870- runit (5.1.7)
2871[2023-07-20T03:22:36+00:00] INFO: Storing updated cookbooks/acme/.rubocop.yml in the cache.
2872[2023-07-20T03:22:36+00:00] INFO: Storing updated cookbooks/acme/metadata.rb in the cache.
2873[2023-07-20T03:22:36+00:00] INFO: Storing updated cookbooks/acme/Berksfile in the cache.
2874[2023-07-20T03:22:36+00:00] INFO: Storing updated cookbooks/crond/templates/default/sv-crond-log-config.erb in the cache.
2875[2023-07-20T03:22:36+00:00] INFO: Storing updated cookbooks/crond/templates/default/sv-crond-log-run.erb in the cache.
2876[2023-07-20T03:22:36+00:00] INFO: Storing updated cookbooks/crond/recipes/disable.rb in the cache.
2877[2023-07-20T03:22:36+00:00] INFO: Storing updated cookbooks/acme/LICENSE in the cache.
2878[2023-07-20T03:22:36+00:00] INFO: Storing updated cookbooks/crond/recipes/enable.rb in the cache.
2879[2023-07-20T03:22:36+00:00] INFO: Storing updated cookbooks/crond/resources/job.rb in the cache.
2880[2023-07-20T03:22:36+00:00] INFO: Storing updated cookbooks/crond/attributes/default.rb in the cache.
2881[2023-07-20T03:22:36+00:00] INFO: Storing updated cookbooks/crond/templates/default/sv-crond-run.erb in the cache.
2882 - acme (4.1.6)
2883[2023-07-20T03:22:36+00:00] INFO: Storing updated cookbooks/crond/metadata.rb in the cache.
2884 - crond (0.1.0)
2885Installing cookbook gem dependencies:
2886Compiling cookbooks...
2887Top level ::CompositeIO is deprecated, require 'multipart/post' and use `Multipart::Post::CompositeReadIO` instead!
2888Top level ::Parts is deprecated, require 'multipart/post' and use `Multipart::Post::Parts` instead!
2889Loading Cinc Auditor profile files:
2890Loading Cinc Auditor input files:
2891Loading Cinc Auditor waiver files:
2892[2023-07-20T03:22:37+00:00] INFO: Generating default secrets
2893[2023-07-20T03:22:38+00:00] INFO: Generating gitlab-secrets.json file
2894Recipe: gitlab::default
2895 * directory[/etc/gitlab] action create[2023-07-20T03:22:38+00:00] INFO: directory[/etc/gitlab] mode changed to 775
2896 - change mode from '0755' to '0775'
2897[2023-07-20T03:22:39+00:00] WARN: gitlab-rails does not have a log_group or default logdir mode defined. Setting to 0700.
2898[2023-07-20T03:22:39+00:00] INFO: Skipped selecting an init system because it was explicitly disabled
2899[2023-07-20T03:22:39+00:00] WARN: gitlab-shell does not have a log_group or default logdir mode defined. Setting to 0700.
2900[2023-07-20T03:22:39+00:00] WARN: gitlab-sshd does not have a log_group or default logdir mode defined. Setting to 0700.
2901[2023-07-20T03:22:39+00:00] WARN: logrotate does not have a log_group or default logdir mode defined. Setting to 0700.
2902[2023-07-20T03:22:39+00:00] WARN: logrotate does not have a log_group or default logdir mode defined. Setting to 0700.
2903[2023-07-20T03:22:39+00:00] WARN: puma does not have a log_group or default logdir mode defined. Setting to 0700.
2904[2023-07-20T03:22:39+00:00] WARN: gitlab-rails does not have a log_group or default logdir mode defined. Setting to 0700.
2905[2023-07-20T03:22:39+00:00] WARN: gitlab-shell does not have a log_group or default logdir mode defined. Setting to 0700.
2906[2023-07-20T03:22:39+00:00] WARN: gitlab-workhorse does not have a log_group or default logdir mode defined. Setting to 0700.
2907[2023-07-20T03:22:39+00:00] WARN: gitlab-pages does not have a log_group or default logdir mode defined. Setting to 0700.
2908[2023-07-20T03:22:39+00:00] WARN: gitlab-kas does not have a log_group or default logdir mode defined. Setting to 0700.
2909[2023-07-20T03:22:39+00:00] WARN: gitaly does not have a log_group or default logdir mode defined. Setting to 0700.
2910[2023-07-20T03:22:39+00:00] WARN: mailroom does not have a log_group or default logdir mode defined. Setting to 0700.
2911[2023-07-20T03:22:39+00:00] WARN: praefect does not have a log_group or default logdir mode defined. Setting to 0700.
2912[2023-07-20T03:22:39+00:00] WARN: praefect does not have a log_group or default logdir mode defined. Setting to 0700.
2913[2023-07-20T03:22:39+00:00] WARN: gitlab-kas does not have a log_group or default logdir mode defined. Setting to 0700.
2914[2023-07-20T03:22:39+00:00] WARN: prometheus does not have a log_group or default logdir mode defined. Setting to 0700.
2915[2023-07-20T03:22:39+00:00] WARN: alertmanager does not have a log_group or default logdir mode defined. Setting to 0700.
2916/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/ohai-17.9.0/lib/ohai/plugins/rpm.rb:25: warning: already initialized constant MACROS_MARKER
2917/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/ohai-17.9.0/lib/ohai/plugins/rpm.rb:25: warning: previous definition of MACROS_MARKER was here
2918/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/ohai-17.9.0/lib/ohai/plugins/rpm.rb:27: warning: already initialized constant DO_NOT_SPLIT
2919/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/ohai-17.9.0/lib/ohai/plugins/rpm.rb:27: warning: previous definition of DO_NOT_SPLIT was here
2920 Converging 210 resources
2921 * directory[/etc/gitlab] action create (up to date)
2922 * directory[Create /var/opt/gitlab] action create[2023-07-20T03:22:39+00:00] INFO: directory[Create /var/opt/gitlab] owner changed to 0
2923[2023-07-20T03:22:39+00:00] INFO: directory[Create /var/opt/gitlab] group changed to 0
2924[2023-07-20T03:22:39+00:00] INFO: directory[Create /var/opt/gitlab] mode changed to 755
2925 - change mode from '0750' to '0755'
2926 - change owner from 'git' to 'root'
2927 - change group from 'git' to 'root'
2928 * directory[Create /var/log/gitlab] action create (up to date)
2929 * directory[/opt/gitlab/embedded/etc] action create[2023-07-20T03:22:39+00:00] INFO: directory[/opt/gitlab/embedded/etc] created directory /opt/gitlab/embedded/etc
2930 - create new directory /opt/gitlab/embedded/etc[2023-07-20T03:22:39+00:00] INFO: directory[/opt/gitlab/embedded/etc] owner changed to 0
2931[2023-07-20T03:22:39+00:00] INFO: directory[/opt/gitlab/embedded/etc] group changed to 0
2932[2023-07-20T03:22:39+00:00] INFO: directory[/opt/gitlab/embedded/etc] mode changed to 755
2933 - change mode from '' to '0755'
2934 - change owner from '' to 'root'
2935 - change group from '' to 'root'
2936 * template[/opt/gitlab/embedded/etc/gitconfig] action create[2023-07-20T03:22:39+00:00] INFO: template[/opt/gitlab/embedded/etc/gitconfig] created file /opt/gitlab/embedded/etc/gitconfig
2937 - create new file /opt/gitlab/embedded/etc/gitconfig[2023-07-20T03:22:39+00:00] INFO: template[/opt/gitlab/embedded/etc/gitconfig] updated file contents /opt/gitlab/embedded/etc/gitconfig
2938 - update content in file /opt/gitlab/embedded/etc/gitconfig from none to e3b0c4
2939 (no diff)[2023-07-20T03:22:39+00:00] INFO: template[/opt/gitlab/embedded/etc/gitconfig] mode changed to 755
2940 - change mode from '' to '0755'
2941Recipe: gitlab::web-server
2942 * account[Webserver user and group] action create (up to date)
2943Recipe: gitlab::users
2944 * directory[/var/opt/gitlab] action create (up to date)
2945 * account[GitLab user and group] action create (up to date)
2946 * template[/var/opt/gitlab/.gitconfig] action create[2023-07-20T03:22:39+00:00] INFO: template[/var/opt/gitlab/.gitconfig] created file /var/opt/gitlab/.gitconfig
2947 - create new file /var/opt/gitlab/.gitconfig[2023-07-20T03:22:39+00:00] INFO: template[/var/opt/gitlab/.gitconfig] updated file contents /var/opt/gitlab/.gitconfig
2948 - update content in file /var/opt/gitlab/.gitconfig from none to 8face2
2949 --- /var/opt/gitlab/.gitconfig 2023-07-20 03:22:39.000000000 +0000
2950 +++ /var/opt/gitlab/.chef-.gitconfig20230720-37-b7urgd.gitconfig 2023-07-20 03:22:39.000000000 +0000
2951 @@ -1,12 +1,24 @@
2952 +# This file is managed by gitlab-ctl. Manual changes will be
2953 +# erased! To change the contents below, edit /etc/gitlab/gitlab.rb
2954 +# and run `sudo gitlab-ctl reconfigure`.
2955 +
2956 +[user]
2957 + name = GitLab
2958 + email = gitlab@praefect.test
2959 +[core]
2960 + autocrlf = input
2961 +
2962 +[gc]
2963 + auto = 0[2023-07-20T03:22:39+00:00] INFO: template[/var/opt/gitlab/.gitconfig] owner changed to 998
2964[2023-07-20T03:22:39+00:00] INFO: template[/var/opt/gitlab/.gitconfig] group changed to 998
2965[2023-07-20T03:22:39+00:00] INFO: template[/var/opt/gitlab/.gitconfig] mode changed to 644
2966 - change mode from '' to '0644'
2967 - change owner from '' to 'git'
2968 - change group from '' to 'git'
2969 * directory[/var/opt/gitlab/.bundle] action create[2023-07-20T03:22:39+00:00] INFO: directory[/var/opt/gitlab/.bundle] created directory /var/opt/gitlab/.bundle
2970 - create new directory /var/opt/gitlab/.bundle[2023-07-20T03:22:39+00:00] INFO: directory[/var/opt/gitlab/.bundle] owner changed to 998
2971[2023-07-20T03:22:39+00:00] INFO: directory[/var/opt/gitlab/.bundle] group changed to 998
2972 - change owner from '' to 'git'
2973 - change group from '' to 'git'
2974Recipe: gitlab::gitlab-rails
2975 * storage_directory[/var/opt/gitlab/git-data] action create
2976 * ruby_block[directory resource: /var/opt/gitlab/git-data] action run[2023-07-20T03:22:39+00:00] INFO: ruby_block[directory resource: /var/opt/gitlab/git-data] called
2977 - execute the ruby block directory resource: /var/opt/gitlab/git-data
2978
2979 * storage_directory[/var/opt/gitlab/git-data/repositories] action create
2980 * ruby_block[directory resource: /var/opt/gitlab/git-data/repositories] action run[2023-07-20T03:22:40+00:00] INFO: ruby_block[directory resource: /var/opt/gitlab/git-data/repositories] called
2981 - execute the ruby block directory resource: /var/opt/gitlab/git-data/repositories
2982
2983Recipe: gitlab::rails_pages_shared_path
2984 * storage_directory[/var/opt/gitlab/gitlab-rails/shared] action create
2985 * ruby_block[directory resource: /var/opt/gitlab/gitlab-rails/shared] action run[2023-07-20T03:22:40+00:00] INFO: ruby_block[directory resource: /var/opt/gitlab/gitlab-rails/shared] called
2986 - execute the ruby block directory resource: /var/opt/gitlab/gitlab-rails/shared
2987
2988 * storage_directory[/var/opt/gitlab/gitlab-rails/shared/pages] action create
2989 * ruby_block[directory resource: /var/opt/gitlab/gitlab-rails/shared/pages] action run[2023-07-20T03:22:40+00:00] INFO: ruby_block[directory resource: /var/opt/gitlab/gitlab-rails/shared/pages] called
2990 - execute the ruby block directory resource: /var/opt/gitlab/gitlab-rails/shared/pages
2991
2992Recipe: gitlab::gitlab-rails
2993 * storage_directory[/var/opt/gitlab/gitlab-rails/shared/artifacts] action create
2994 * ruby_block[directory resource: /var/opt/gitlab/gitlab-rails/shared/artifacts] action run[2023-07-20T03:22:40+00:00] INFO: ruby_block[directory resource: /var/opt/gitlab/gitlab-rails/shared/artifacts] called
2995 - execute the ruby block directory resource: /var/opt/gitlab/gitlab-rails/shared/artifacts
2996
2997 * storage_directory[/var/opt/gitlab/gitlab-rails/shared/external-diffs] action create
2998 * ruby_block[directory resource: /var/opt/gitlab/gitlab-rails/shared/external-diffs] action run[2023-07-20T03:22:40+00:00] INFO: ruby_block[directory resource: /var/opt/gitlab/gitlab-rails/shared/external-diffs] called
2999 - execute the ruby block directory resource: /var/opt/gitlab/gitlab-rails/shared/external-diffs
3000
3001 * storage_directory[/var/opt/gitlab/gitlab-rails/shared/lfs-objects] action create
3002 * ruby_block[directory resource: /var/opt/gitlab/gitlab-rails/shared/lfs-objects] action run[2023-07-20T03:22:41+00:00] INFO: ruby_block[directory resource: /var/opt/gitlab/gitlab-rails/shared/lfs-objects] called
3003 - execute the ruby block directory resource: /var/opt/gitlab/gitlab-rails/shared/lfs-objects
3004
3005 * storage_directory[/var/opt/gitlab/gitlab-rails/shared/packages] action create
3006 * ruby_block[directory resource: /var/opt/gitlab/gitlab-rails/shared/packages] action run[2023-07-20T03:22:41+00:00] INFO: ruby_block[directory resource: /var/opt/gitlab/gitlab-rails/shared/packages] called
3007 - execute the ruby block directory resource: /var/opt/gitlab/gitlab-rails/shared/packages
3008
3009 * storage_directory[/var/opt/gitlab/gitlab-rails/shared/dependency_proxy] action create
3010 * ruby_block[directory resource: /var/opt/gitlab/gitlab-rails/shared/dependency_proxy] action run[2023-07-20T03:22:41+00:00] INFO: ruby_block[directory resource: /var/opt/gitlab/gitlab-rails/shared/dependency_proxy] called
3011 - execute the ruby block directory resource: /var/opt/gitlab/gitlab-rails/shared/dependency_proxy
3012
3013 * storage_directory[/var/opt/gitlab/gitlab-rails/shared/terraform_state] action create
3014 * ruby_block[directory resource: /var/opt/gitlab/gitlab-rails/shared/terraform_state] action run[2023-07-20T03:22:41+00:00] INFO: ruby_block[directory resource: /var/opt/gitlab/gitlab-rails/shared/terraform_state] called
3015 - execute the ruby block directory resource: /var/opt/gitlab/gitlab-rails/shared/terraform_state
3016
3017 * storage_directory[/var/opt/gitlab/gitlab-rails/shared/ci_secure_files] action create
3018 * ruby_block[directory resource: /var/opt/gitlab/gitlab-rails/shared/ci_secure_files] action run[2023-07-20T03:22:41+00:00] INFO: ruby_block[directory resource: /var/opt/gitlab/gitlab-rails/shared/ci_secure_files] called
3019 - execute the ruby block directory resource: /var/opt/gitlab/gitlab-rails/shared/ci_secure_files
3020
3021 * storage_directory[/var/opt/gitlab/gitlab-rails/shared/encrypted_settings] action create
3022 * ruby_block[directory resource: /var/opt/gitlab/gitlab-rails/shared/encrypted_settings] action run[2023-07-20T03:22:42+00:00] INFO: ruby_block[directory resource: /var/opt/gitlab/gitlab-rails/shared/encrypted_settings] called
3023 - execute the ruby block directory resource: /var/opt/gitlab/gitlab-rails/shared/encrypted_settings
3024
3025 * storage_directory[/var/opt/gitlab/gitlab-rails/uploads] action create
3026 * ruby_block[directory resource: /var/opt/gitlab/gitlab-rails/uploads] action run[2023-07-20T03:22:42+00:00] INFO: ruby_block[directory resource: /var/opt/gitlab/gitlab-rails/uploads] called
3027 - execute the ruby block directory resource: /var/opt/gitlab/gitlab-rails/uploads
3028
3029 * storage_directory[/var/opt/gitlab/gitlab-ci/builds] action create
3030 * ruby_block[directory resource: /var/opt/gitlab/gitlab-ci/builds] action run[2023-07-20T03:22:42+00:00] INFO: ruby_block[directory resource: /var/opt/gitlab/gitlab-ci/builds] called
3031 - execute the ruby block directory resource: /var/opt/gitlab/gitlab-ci/builds
3032
3033 * storage_directory[/var/opt/gitlab/gitlab-rails/shared/cache] action create
3034 * ruby_block[directory resource: /var/opt/gitlab/gitlab-rails/shared/cache] action run[2023-07-20T03:22:42+00:00] INFO: ruby_block[directory resource: /var/opt/gitlab/gitlab-rails/shared/cache] called
3035 - execute the ruby block directory resource: /var/opt/gitlab/gitlab-rails/shared/cache
3036
3037 * storage_directory[/var/opt/gitlab/gitlab-rails/shared/tmp] action create
3038 * ruby_block[directory resource: /var/opt/gitlab/gitlab-rails/shared/tmp] action run[2023-07-20T03:22:42+00:00] INFO: ruby_block[directory resource: /var/opt/gitlab/gitlab-rails/shared/tmp] called
3039 - execute the ruby block directory resource: /var/opt/gitlab/gitlab-rails/shared/tmp
3040
3041 * storage_directory[/opt/gitlab/embedded/service/gitlab-rails/public] action create (skipped due to only_if)
3042 * directory[create /var/opt/gitlab/gitlab-rails/etc] action create[2023-07-20T03:22:43+00:00] INFO: directory[create /var/opt/gitlab/gitlab-rails/etc] created directory /var/opt/gitlab/gitlab-rails/etc
3043 - create new directory /var/opt/gitlab/gitlab-rails/etc[2023-07-20T03:22:43+00:00] INFO: directory[create /var/opt/gitlab/gitlab-rails/etc] owner changed to 998
3044[2023-07-20T03:22:43+00:00] INFO: directory[create /var/opt/gitlab/gitlab-rails/etc] mode changed to 700
3045 - change mode from '' to '0700'
3046 - change owner from '' to 'git'
3047 * directory[create /opt/gitlab/etc/gitlab-rails] action create[2023-07-20T03:22:43+00:00] INFO: directory[create /opt/gitlab/etc/gitlab-rails] created directory /opt/gitlab/etc/gitlab-rails
3048 - create new directory /opt/gitlab/etc/gitlab-rails[2023-07-20T03:22:43+00:00] INFO: directory[create /opt/gitlab/etc/gitlab-rails] owner changed to 998
3049[2023-07-20T03:22:43+00:00] INFO: directory[create /opt/gitlab/etc/gitlab-rails] mode changed to 700
3050 - change mode from '' to '0700'
3051 - change owner from '' to 'git'
3052 * directory[create /var/opt/gitlab/gitlab-rails/working] action create[2023-07-20T03:22:43+00:00] INFO: directory[create /var/opt/gitlab/gitlab-rails/working] created directory /var/opt/gitlab/gitlab-rails/working
3053 - create new directory /var/opt/gitlab/gitlab-rails/working[2023-07-20T03:22:43+00:00] INFO: directory[create /var/opt/gitlab/gitlab-rails/working] owner changed to 998
3054[2023-07-20T03:22:43+00:00] INFO: directory[create /var/opt/gitlab/gitlab-rails/working] mode changed to 700
3055 - change mode from '' to '0700'
3056 - change owner from '' to 'git'
3057 * directory[create /var/opt/gitlab/gitlab-rails/tmp] action create[2023-07-20T03:22:43+00:00] INFO: directory[create /var/opt/gitlab/gitlab-rails/tmp] created directory /var/opt/gitlab/gitlab-rails/tmp
3058 - create new directory /var/opt/gitlab/gitlab-rails/tmp[2023-07-20T03:22:43+00:00] INFO: directory[create /var/opt/gitlab/gitlab-rails/tmp] owner changed to 998
3059[2023-07-20T03:22:43+00:00] INFO: directory[create /var/opt/gitlab/gitlab-rails/tmp] mode changed to 700
3060 - change mode from '' to '0700'
3061 - change owner from '' to 'git'
3062 * directory[create /var/opt/gitlab/gitlab-rails/upgrade-status] action create[2023-07-20T03:22:43+00:00] INFO: directory[create /var/opt/gitlab/gitlab-rails/upgrade-status] created directory /var/opt/gitlab/gitlab-rails/upgrade-status
3063 - create new directory /var/opt/gitlab/gitlab-rails/upgrade-status[2023-07-20T03:22:43+00:00] INFO: directory[create /var/opt/gitlab/gitlab-rails/upgrade-status] owner changed to 998
3064[2023-07-20T03:22:43+00:00] INFO: directory[create /var/opt/gitlab/gitlab-rails/upgrade-status] mode changed to 700
3065 - change mode from '' to '0700'
3066 - change owner from '' to 'git'
3067 * directory[/var/log/gitlab/gitlab-rails] action create[2023-07-20T03:22:43+00:00] INFO: directory[/var/log/gitlab/gitlab-rails] created directory /var/log/gitlab/gitlab-rails
3068 - create new directory /var/log/gitlab/gitlab-rails[2023-07-20T03:22:43+00:00] INFO: directory[/var/log/gitlab/gitlab-rails] owner changed to 998
3069[2023-07-20T03:22:43+00:00] INFO: directory[/var/log/gitlab/gitlab-rails] mode changed to 700
3070 - change mode from '' to '0700'
3071 - change owner from '' to 'git'
3072 * storage_directory[/var/opt/gitlab/backups] action create
3073 * ruby_block[directory resource: /var/opt/gitlab/backups] action run[2023-07-20T03:22:43+00:00] INFO: ruby_block[directory resource: /var/opt/gitlab/backups] called
3074 - execute the ruby block directory resource: /var/opt/gitlab/backups
3075
3076 * directory[/var/opt/gitlab/gitlab-rails] action create[2023-07-20T03:22:43+00:00] INFO: directory[/var/opt/gitlab/gitlab-rails] owner changed to 998
3077 - change owner from 'root' to 'git'
3078 * directory[/var/opt/gitlab/gitlab-ci] action create[2023-07-20T03:22:43+00:00] INFO: directory[/var/opt/gitlab/gitlab-ci] owner changed to 998
3079 - change owner from 'root' to 'git'
3080 * file[/var/opt/gitlab/gitlab-rails/etc/gitlab-registry.key] action create (skipped due to only_if)
3081 * template[/opt/gitlab/etc/gitlab-rails-rc] action create[2023-07-20T03:22:43+00:00] INFO: template[/opt/gitlab/etc/gitlab-rails-rc] created file /opt/gitlab/etc/gitlab-rails-rc
3082 - create new file /opt/gitlab/etc/gitlab-rails-rc[2023-07-20T03:22:43+00:00] INFO: template[/opt/gitlab/etc/gitlab-rails-rc] updated file contents /opt/gitlab/etc/gitlab-rails-rc
3083 - update content in file /opt/gitlab/etc/gitlab-rails-rc from none to 7b16c8
3084 --- /opt/gitlab/etc/gitlab-rails-rc 2023-07-20 03:22:43.000000000 +0000
3085 +++ /opt/gitlab/etc/.chef-gitlab-rails-rc20230720-37-r54nrk 2023-07-20 03:22:43.000000000 +0000
3086 @@ -1,5 +1,10 @@
3087 +gitlab_user='git'
3088 +gitlab_group='git'
3089 +registry_dir=''
3090 +registry_user='registry'
3091 +registry_group='registry'[2023-07-20T03:22:43+00:00] INFO: template[/opt/gitlab/etc/gitlab-rails-rc] owner changed to 0
3092[2023-07-20T03:22:43+00:00] INFO: template[/opt/gitlab/etc/gitlab-rails-rc] group changed to 0
3093[2023-07-20T03:22:43+00:00] INFO: template[/opt/gitlab/etc/gitlab-rails-rc] mode changed to 644
3094 - change mode from '' to '0644'
3095 - change owner from '' to 'root'
3096 - change group from '' to 'root'
3097 * file[/opt/gitlab/embedded/service/gitlab-rails/.secret] action delete (up to date)
3098 * file[/var/opt/gitlab/gitlab-rails/etc/secret] action delete (up to date)
3099 * templatesymlink[Create a database.yml and create a symlink to Rails root] action create
3100 * template[/var/opt/gitlab/gitlab-rails/etc/database.yml] action create[2023-07-20T03:22:43+00:00] INFO: template[/var/opt/gitlab/gitlab-rails/etc/database.yml] created file /var/opt/gitlab/gitlab-rails/etc/database.yml
3101 - create new file /var/opt/gitlab/gitlab-rails/etc/database.yml[2023-07-20T03:22:43+00:00] INFO: template[/var/opt/gitlab/gitlab-rails/etc/database.yml] updated file contents /var/opt/gitlab/gitlab-rails/etc/database.yml
3102 - update content in file /var/opt/gitlab/gitlab-rails/etc/database.yml from none to 9c495c
3103 - suppressed sensitive resource[2023-07-20T03:22:43+00:00] INFO: template[/var/opt/gitlab/gitlab-rails/etc/database.yml] owner changed to 0
3104[2023-07-20T03:22:43+00:00] INFO: template[/var/opt/gitlab/gitlab-rails/etc/database.yml] group changed to 998
3105[2023-07-20T03:22:43+00:00] INFO: template[/var/opt/gitlab/gitlab-rails/etc/database.yml] mode changed to 640
3106 - change mode from '' to '0640'
3107 - change owner from '' to 'root'
3108 - change group from '' to 'git'
3109 * link[Link /opt/gitlab/embedded/service/gitlab-rails/config/database.yml to /var/opt/gitlab/gitlab-rails/etc/database.yml] action create[2023-07-20T03:22:43+00:00] INFO: link[Link /opt/gitlab/embedded/service/gitlab-rails/config/database.yml to /var/opt/gitlab/gitlab-rails/etc/database.yml] created
3110 - create symlink at /opt/gitlab/embedded/service/gitlab-rails/config/database.yml to /var/opt/gitlab/gitlab-rails/etc/database.yml
3111
3112 * templatesymlink[Create a secrets.yml and create a symlink to Rails root] action create
3113 * template[/var/opt/gitlab/gitlab-rails/etc/secrets.yml] action create[2023-07-20T03:22:43+00:00] INFO: template[/var/opt/gitlab/gitlab-rails/etc/secrets.yml] created file /var/opt/gitlab/gitlab-rails/etc/secrets.yml
3114 - create new file /var/opt/gitlab/gitlab-rails/etc/secrets.yml[2023-07-20T03:22:43+00:00] INFO: template[/var/opt/gitlab/gitlab-rails/etc/secrets.yml] updated file contents /var/opt/gitlab/gitlab-rails/etc/secrets.yml
3115 - update content in file /var/opt/gitlab/gitlab-rails/etc/secrets.yml from none to 993886
3116 - suppressed sensitive resource[2023-07-20T03:22:43+00:00] INFO: template[/var/opt/gitlab/gitlab-rails/etc/secrets.yml] owner changed to 0
3117[2023-07-20T03:22:43+00:00] INFO: template[/var/opt/gitlab/gitlab-rails/etc/secrets.yml] group changed to 0
3118[2023-07-20T03:22:43+00:00] INFO: template[/var/opt/gitlab/gitlab-rails/etc/secrets.yml] mode changed to 644
3119 - change mode from '' to '0644'
3120 - change owner from '' to 'root'
3121 - change group from '' to 'root'
3122 * link[Link /opt/gitlab/embedded/service/gitlab-rails/config/secrets.yml to /var/opt/gitlab/gitlab-rails/etc/secrets.yml] action create[2023-07-20T03:22:43+00:00] INFO: link[Link /opt/gitlab/embedded/service/gitlab-rails/config/secrets.yml to /var/opt/gitlab/gitlab-rails/etc/secrets.yml] created
3123 - create symlink at /opt/gitlab/embedded/service/gitlab-rails/config/secrets.yml to /var/opt/gitlab/gitlab-rails/etc/secrets.yml
3124
3125 * templatesymlink[Create a resque.yml and create a symlink to Rails root] action create
3126 * template[/var/opt/gitlab/gitlab-rails/etc/resque.yml] action create[2023-07-20T03:22:43+00:00] INFO: template[/var/opt/gitlab/gitlab-rails/etc/resque.yml] created file /var/opt/gitlab/gitlab-rails/etc/resque.yml
3127 - create new file /var/opt/gitlab/gitlab-rails/etc/resque.yml[2023-07-20T03:22:43+00:00] INFO: template[/var/opt/gitlab/gitlab-rails/etc/resque.yml] updated file contents /var/opt/gitlab/gitlab-rails/etc/resque.yml
3128 - update content in file /var/opt/gitlab/gitlab-rails/etc/resque.yml from none to ec4232
3129 - suppressed sensitive resource[2023-07-20T03:22:43+00:00] INFO: template[/var/opt/gitlab/gitlab-rails/etc/resque.yml] owner changed to 0
3130[2023-07-20T03:22:43+00:00] INFO: template[/var/opt/gitlab/gitlab-rails/etc/resque.yml] group changed to 0
3131[2023-07-20T03:22:43+00:00] INFO: template[/var/opt/gitlab/gitlab-rails/etc/resque.yml] mode changed to 644
3132 - change mode from '' to '0644'
3133 - change owner from '' to 'root'
3134 - change group from '' to 'root'
3135 * link[Link /opt/gitlab/embedded/service/gitlab-rails/config/resque.yml to /var/opt/gitlab/gitlab-rails/etc/resque.yml] action create[2023-07-20T03:22:43+00:00] INFO: link[Link /opt/gitlab/embedded/service/gitlab-rails/config/resque.yml to /var/opt/gitlab/gitlab-rails/etc/resque.yml] created
3136 - create symlink at /opt/gitlab/embedded/service/gitlab-rails/config/resque.yml to /var/opt/gitlab/gitlab-rails/etc/resque.yml
3137
3138 * templatesymlink[Create an override redis.yml and create a symlink to Rails root] action create
3139 * template[/var/opt/gitlab/gitlab-rails/etc/redis.yml] action create[2023-07-20T03:22:43+00:00] INFO: template[/var/opt/gitlab/gitlab-rails/etc/redis.yml] created file /var/opt/gitlab/gitlab-rails/etc/redis.yml
3140 - create new file /var/opt/gitlab/gitlab-rails/etc/redis.yml[2023-07-20T03:22:43+00:00] INFO: template[/var/opt/gitlab/gitlab-rails/etc/redis.yml] updated file contents /var/opt/gitlab/gitlab-rails/etc/redis.yml
3141 - update content in file /var/opt/gitlab/gitlab-rails/etc/redis.yml from none to e3b0c4
3142 - suppressed sensitive resource[2023-07-20T03:22:43+00:00] INFO: template[/var/opt/gitlab/gitlab-rails/etc/redis.yml] owner changed to 0
3143[2023-07-20T03:22:43+00:00] INFO: template[/var/opt/gitlab/gitlab-rails/etc/redis.yml] group changed to 0
3144[2023-07-20T03:22:43+00:00] INFO: template[/var/opt/gitlab/gitlab-rails/etc/redis.yml] mode changed to 644
3145 - change mode from '' to '0644'
3146 - change owner from '' to 'root'
3147 - change group from '' to 'root'
3148 * link[Link /opt/gitlab/embedded/service/gitlab-rails/config/redis.yml to /var/opt/gitlab/gitlab-rails/etc/redis.yml] action create[2023-07-20T03:22:43+00:00] INFO: link[Link /opt/gitlab/embedded/service/gitlab-rails/config/redis.yml to /var/opt/gitlab/gitlab-rails/etc/redis.yml] created
3149 - create symlink at /opt/gitlab/embedded/service/gitlab-rails/config/redis.yml to /var/opt/gitlab/gitlab-rails/etc/redis.yml
3150
3151 * templatesymlink[Create a cable.yml and create a symlink to Rails root] action create
3152 * template[/var/opt/gitlab/gitlab-rails/etc/cable.yml] action create[2023-07-20T03:22:43+00:00] INFO: template[/var/opt/gitlab/gitlab-rails/etc/cable.yml] created file /var/opt/gitlab/gitlab-rails/etc/cable.yml
3153 - create new file /var/opt/gitlab/gitlab-rails/etc/cable.yml[2023-07-20T03:22:43+00:00] INFO: template[/var/opt/gitlab/gitlab-rails/etc/cable.yml] updated file contents /var/opt/gitlab/gitlab-rails/etc/cable.yml
3154 - update content in file /var/opt/gitlab/gitlab-rails/etc/cable.yml from none to 551667
3155 - suppressed sensitive resource[2023-07-20T03:22:43+00:00] INFO: template[/var/opt/gitlab/gitlab-rails/etc/cable.yml] owner changed to 0
3156[2023-07-20T03:22:43+00:00] INFO: template[/var/opt/gitlab/gitlab-rails/etc/cable.yml] group changed to 0
3157[2023-07-20T03:22:43+00:00] INFO: template[/var/opt/gitlab/gitlab-rails/etc/cable.yml] mode changed to 644
3158 - change mode from '' to '0644'
3159 - change owner from '' to 'root'
3160 - change group from '' to 'root'
3161 * link[Link /opt/gitlab/embedded/service/gitlab-rails/config/cable.yml to /var/opt/gitlab/gitlab-rails/etc/cable.yml] action create[2023-07-20T03:22:43+00:00] INFO: link[Link /opt/gitlab/embedded/service/gitlab-rails/config/cable.yml to /var/opt/gitlab/gitlab-rails/etc/cable.yml] created
3162 - create symlink at /opt/gitlab/embedded/service/gitlab-rails/config/cable.yml to /var/opt/gitlab/gitlab-rails/etc/cable.yml
3163
3164 * templatesymlink[Create a redis.cache.yml and create a symlink to Rails root] action delete
3165 * file[/var/opt/gitlab/gitlab-rails/etc/redis.cache.yml] action delete (up to date)
3166 * link[/opt/gitlab/embedded/service/gitlab-rails/config/redis.cache.yml] action delete (up to date)
3167 (up to date)
3168 * templatesymlink[Create a redis.queues.yml and create a symlink to Rails root] action delete
3169 * file[/var/opt/gitlab/gitlab-rails/etc/redis.queues.yml] action delete (up to date)
3170 * link[/opt/gitlab/embedded/service/gitlab-rails/config/redis.queues.yml] action delete (up to date)
3171 (up to date)
3172 * templatesymlink[Create a redis.shared_state.yml and create a symlink to Rails root] action delete
3173 * file[/var/opt/gitlab/gitlab-rails/etc/redis.shared_state.yml] action delete (up to date)
3174 * link[/opt/gitlab/embedded/service/gitlab-rails/config/redis.shared_state.yml] action delete (up to date)
3175 (up to date)
3176 * templatesymlink[Create a redis.trace_chunks.yml and create a symlink to Rails root] action delete
3177 * file[/var/opt/gitlab/gitlab-rails/etc/redis.trace_chunks.yml] action delete (up to date)
3178 * link[/opt/gitlab/embedded/service/gitlab-rails/config/redis.trace_chunks.yml] action delete (up to date)
3179 (up to date)
3180 * templatesymlink[Create a redis.rate_limiting.yml and create a symlink to Rails root] action delete
3181 * file[/var/opt/gitlab/gitlab-rails/etc/redis.rate_limiting.yml] action delete (up to date)
3182 * link[/opt/gitlab/embedded/service/gitlab-rails/config/redis.rate_limiting.yml] action delete (up to date)
3183 (up to date)
3184 * templatesymlink[Create a redis.sessions.yml and create a symlink to Rails root] action delete
3185 * file[/var/opt/gitlab/gitlab-rails/etc/redis.sessions.yml] action delete (up to date)
3186 * link[/opt/gitlab/embedded/service/gitlab-rails/config/redis.sessions.yml] action delete (up to date)
3187 (up to date)
3188 * templatesymlink[Create a redis.repository_cache.yml and create a symlink to Rails root] action delete
3189 * file[/var/opt/gitlab/gitlab-rails/etc/redis.repository_cache.yml] action delete (up to date)
3190 * link[/opt/gitlab/embedded/service/gitlab-rails/config/redis.repository_cache.yml] action delete (up to date)
3191 (up to date)
3192 * templatesymlink[Create a redis.cluster_rate_limiting.yml and create a symlink to Rails root] action delete
3193 * file[/var/opt/gitlab/gitlab-rails/etc/redis.cluster_rate_limiting.yml] action delete (up to date)
3194 * link[/opt/gitlab/embedded/service/gitlab-rails/config/redis.cluster_rate_limiting.yml] action delete (up to date)
3195 (up to date)
3196 * templatesymlink[Create a smtp_settings.rb and create a symlink to Rails root] action delete
3197 * file[/var/opt/gitlab/gitlab-rails/etc/smtp_settings.rb] action delete (up to date)
3198 * link[/opt/gitlab/embedded/service/gitlab-rails/config/initializers/smtp_settings.rb] action delete (up to date)
3199 (up to date)
3200 * templatesymlink[Create a gitlab.yml and create a symlink to Rails root] action create
3201 * template[/var/opt/gitlab/gitlab-rails/etc/gitlab.yml] action create[2023-07-20T03:22:43+00:00] INFO: template[/var/opt/gitlab/gitlab-rails/etc/gitlab.yml] created file /var/opt/gitlab/gitlab-rails/etc/gitlab.yml
3202 - create new file /var/opt/gitlab/gitlab-rails/etc/gitlab.yml[2023-07-20T03:22:43+00:00] INFO: template[/var/opt/gitlab/gitlab-rails/etc/gitlab.yml] updated file contents /var/opt/gitlab/gitlab-rails/etc/gitlab.yml
3203 - update content in file /var/opt/gitlab/gitlab-rails/etc/gitlab.yml from none to ff2b6e
3204 - suppressed sensitive resource[2023-07-20T03:22:43+00:00] INFO: template[/var/opt/gitlab/gitlab-rails/etc/gitlab.yml] owner changed to 0
3205[2023-07-20T03:22:43+00:00] INFO: template[/var/opt/gitlab/gitlab-rails/etc/gitlab.yml] group changed to 998
3206[2023-07-20T03:22:43+00:00] INFO: template[/var/opt/gitlab/gitlab-rails/etc/gitlab.yml] mode changed to 640
3207 - change mode from '' to '0640'
3208 - change owner from '' to 'root'
3209 - change group from '' to 'git'
3210 * link[Link /opt/gitlab/embedded/service/gitlab-rails/config/gitlab.yml to /var/opt/gitlab/gitlab-rails/etc/gitlab.yml] action create[2023-07-20T03:22:43+00:00] INFO: link[Link /opt/gitlab/embedded/service/gitlab-rails/config/gitlab.yml to /var/opt/gitlab/gitlab-rails/etc/gitlab.yml] created
3211 - create symlink at /opt/gitlab/embedded/service/gitlab-rails/config/gitlab.yml to /var/opt/gitlab/gitlab-rails/etc/gitlab.yml
3212
3213 * templatesymlink[Create a gitlab_workhorse_secret and create a symlink to Rails root] action create
3214 * template[/var/opt/gitlab/gitlab-rails/etc/gitlab_workhorse_secret] action create[2023-07-20T03:22:43+00:00] INFO: template[/var/opt/gitlab/gitlab-rails/etc/gitlab_workhorse_secret] created file /var/opt/gitlab/gitlab-rails/etc/gitlab_workhorse_secret
3215 - create new file /var/opt/gitlab/gitlab-rails/etc/gitlab_workhorse_secret[2023-07-20T03:22:43+00:00] INFO: template[/var/opt/gitlab/gitlab-rails/etc/gitlab_workhorse_secret] updated file contents /var/opt/gitlab/gitlab-rails/etc/gitlab_workhorse_secret
3216 - update content in file /var/opt/gitlab/gitlab-rails/etc/gitlab_workhorse_secret from none to 5e89a1
3217 - suppressed sensitive resource[2023-07-20T03:22:43+00:00] INFO: template[/var/opt/gitlab/gitlab-rails/etc/gitlab_workhorse_secret] owner changed to 0
3218[2023-07-20T03:22:43+00:00] INFO: template[/var/opt/gitlab/gitlab-rails/etc/gitlab_workhorse_secret] group changed to 0
3219[2023-07-20T03:22:43+00:00] INFO: template[/var/opt/gitlab/gitlab-rails/etc/gitlab_workhorse_secret] mode changed to 644
3220 - change mode from '' to '0644'
3221 - change owner from '' to 'root'
3222 - change group from '' to 'root'
3223 * link[Link /opt/gitlab/embedded/service/gitlab-rails/.gitlab_workhorse_secret to /var/opt/gitlab/gitlab-rails/etc/gitlab_workhorse_secret] action create[2023-07-20T03:22:43+00:00] INFO: link[Link /opt/gitlab/embedded/service/gitlab-rails/.gitlab_workhorse_secret to /var/opt/gitlab/gitlab-rails/etc/gitlab_workhorse_secret] created
3224 - create symlink at /opt/gitlab/embedded/service/gitlab-rails/.gitlab_workhorse_secret to /var/opt/gitlab/gitlab-rails/etc/gitlab_workhorse_secret
3225
3226 * templatesymlink[Create a gitlab_shell_secret and create a symlink to Rails root] action create
3227 * template[/var/opt/gitlab/gitlab-rails/etc/gitlab_shell_secret] action create[2023-07-20T03:22:43+00:00] INFO: template[/var/opt/gitlab/gitlab-rails/etc/gitlab_shell_secret] created file /var/opt/gitlab/gitlab-rails/etc/gitlab_shell_secret
3228 - create new file /var/opt/gitlab/gitlab-rails/etc/gitlab_shell_secret[2023-07-20T03:22:43+00:00] INFO: template[/var/opt/gitlab/gitlab-rails/etc/gitlab_shell_secret] updated file contents /var/opt/gitlab/gitlab-rails/etc/gitlab_shell_secret
3229 - update content in file /var/opt/gitlab/gitlab-rails/etc/gitlab_shell_secret from none to 158e56
3230 - suppressed sensitive resource[2023-07-20T03:22:43+00:00] INFO: template[/var/opt/gitlab/gitlab-rails/etc/gitlab_shell_secret] owner changed to 0
3231[2023-07-20T03:22:43+00:00] INFO: template[/var/opt/gitlab/gitlab-rails/etc/gitlab_shell_secret] group changed to 0
3232[2023-07-20T03:22:43+00:00] INFO: template[/var/opt/gitlab/gitlab-rails/etc/gitlab_shell_secret] mode changed to 644
3233 - change mode from '' to '0644'
3234 - change owner from '' to 'root'
3235 - change group from '' to 'root'
3236 * link[Link /opt/gitlab/embedded/service/gitlab-rails/.gitlab_shell_secret to /var/opt/gitlab/gitlab-rails/etc/gitlab_shell_secret] action create[2023-07-20T03:22:43+00:00] INFO: link[Link /opt/gitlab/embedded/service/gitlab-rails/.gitlab_shell_secret to /var/opt/gitlab/gitlab-rails/etc/gitlab_shell_secret] created
3237 - create symlink at /opt/gitlab/embedded/service/gitlab-rails/.gitlab_shell_secret to /var/opt/gitlab/gitlab-rails/etc/gitlab_shell_secret
3238
3239 * templatesymlink[Create a gitlab_incoming_email_secret and create a symlink to Rails root] action create (skipped due to only_if)
3240 * templatesymlink[Create a gitlab_service_desk_email_secret and create a symlink to Rails root] action create (skipped due to only_if)
3241 * templatesymlink[Create a gitlab_pages_secret and create a symlink to Rails root] action create[2023-07-20T03:22:43+00:00] WARN: only_if block for templatesymlink[Create a gitlab_pages_secret and create a symlink to Rails root] returned a string, did you mean to run a command?
3242 * template[/var/opt/gitlab/gitlab-rails/etc/gitlab_pages_secret] action create[2023-07-20T03:22:43+00:00] INFO: template[/var/opt/gitlab/gitlab-rails/etc/gitlab_pages_secret] created file /var/opt/gitlab/gitlab-rails/etc/gitlab_pages_secret
3243 - create new file /var/opt/gitlab/gitlab-rails/etc/gitlab_pages_secret[2023-07-20T03:22:43+00:00] INFO: template[/var/opt/gitlab/gitlab-rails/etc/gitlab_pages_secret] updated file contents /var/opt/gitlab/gitlab-rails/etc/gitlab_pages_secret
3244 - update content in file /var/opt/gitlab/gitlab-rails/etc/gitlab_pages_secret from none to 7a6366
3245 - suppressed sensitive resource[2023-07-20T03:22:43+00:00] INFO: template[/var/opt/gitlab/gitlab-rails/etc/gitlab_pages_secret] owner changed to 0
3246[2023-07-20T03:22:43+00:00] INFO: template[/var/opt/gitlab/gitlab-rails/etc/gitlab_pages_secret] group changed to 0
3247[2023-07-20T03:22:43+00:00] INFO: template[/var/opt/gitlab/gitlab-rails/etc/gitlab_pages_secret] mode changed to 644
3248 - change mode from '' to '0644'
3249 - change owner from '' to 'root'
3250 - change group from '' to 'root'
3251 * link[Link /opt/gitlab/embedded/service/gitlab-rails/.gitlab_pages_secret to /var/opt/gitlab/gitlab-rails/etc/gitlab_pages_secret] action create[2023-07-20T03:22:43+00:00] INFO: link[Link /opt/gitlab/embedded/service/gitlab-rails/.gitlab_pages_secret to /var/opt/gitlab/gitlab-rails/etc/gitlab_pages_secret] created
3252 - create symlink at /opt/gitlab/embedded/service/gitlab-rails/.gitlab_pages_secret to /var/opt/gitlab/gitlab-rails/etc/gitlab_pages_secret
3253
3254 * templatesymlink[Create a gitlab_kas_secret and create a symlink to Rails root] action create[2023-07-20T03:22:43+00:00] WARN: only_if block for templatesymlink[Create a gitlab_kas_secret and create a symlink to Rails root] returned a string, did you mean to run a command?
3255 * template[/var/opt/gitlab/gitlab-rails/etc/gitlab_kas_secret] action create[2023-07-20T03:22:43+00:00] INFO: template[/var/opt/gitlab/gitlab-rails/etc/gitlab_kas_secret] created file /var/opt/gitlab/gitlab-rails/etc/gitlab_kas_secret
3256 - create new file /var/opt/gitlab/gitlab-rails/etc/gitlab_kas_secret[2023-07-20T03:22:43+00:00] INFO: template[/var/opt/gitlab/gitlab-rails/etc/gitlab_kas_secret] updated file contents /var/opt/gitlab/gitlab-rails/etc/gitlab_kas_secret
3257 - update content in file /var/opt/gitlab/gitlab-rails/etc/gitlab_kas_secret from none to db4b89
3258 - suppressed sensitive resource[2023-07-20T03:22:43+00:00] INFO: template[/var/opt/gitlab/gitlab-rails/etc/gitlab_kas_secret] owner changed to 0
3259[2023-07-20T03:22:43+00:00] INFO: template[/var/opt/gitlab/gitlab-rails/etc/gitlab_kas_secret] group changed to 0
3260[2023-07-20T03:22:43+00:00] INFO: template[/var/opt/gitlab/gitlab-rails/etc/gitlab_kas_secret] mode changed to 644
3261 - change mode from '' to '0644'
3262 - change owner from '' to 'root'
3263 - change group from '' to 'root'
3264 * link[Link /opt/gitlab/embedded/service/gitlab-rails/.gitlab_kas_secret to /var/opt/gitlab/gitlab-rails/etc/gitlab_kas_secret] action create[2023-07-20T03:22:43+00:00] INFO: link[Link /opt/gitlab/embedded/service/gitlab-rails/.gitlab_kas_secret to /var/opt/gitlab/gitlab-rails/etc/gitlab_kas_secret] created
3265 - create symlink at /opt/gitlab/embedded/service/gitlab-rails/.gitlab_kas_secret to /var/opt/gitlab/gitlab-rails/etc/gitlab_kas_secret
3266
3267 * link[/opt/gitlab/embedded/service/gitlab-rails/config/initializers/relative_url.rb] action delete (up to date)
3268 * file[/var/opt/gitlab/gitlab-rails/etc/relative_url.rb] action delete (up to date)
3269 * env_dir[/opt/gitlab/etc/gitlab-rails/env] action create
3270 * directory[/opt/gitlab/etc/gitlab-rails/env] action create[2023-07-20T03:22:43+00:00] INFO: directory[/opt/gitlab/etc/gitlab-rails/env] created directory /opt/gitlab/etc/gitlab-rails/env
3271 - create new directory /opt/gitlab/etc/gitlab-rails/env
3272 * file[/opt/gitlab/etc/gitlab-rails/env/HOME] action create[2023-07-20T03:22:43+00:00] INFO: file[/opt/gitlab/etc/gitlab-rails/env/HOME] created file /opt/gitlab/etc/gitlab-rails/env/HOME
3273 - create new file /opt/gitlab/etc/gitlab-rails/env/HOME[2023-07-20T03:22:43+00:00] INFO: file[/opt/gitlab/etc/gitlab-rails/env/HOME] updated file contents /opt/gitlab/etc/gitlab-rails/env/HOME
3274 - update content in file /opt/gitlab/etc/gitlab-rails/env/HOME from none to 205bb9
3275 - suppressed sensitive resource
3276 * file[/opt/gitlab/etc/gitlab-rails/env/RAILS_ENV] action create[2023-07-20T03:22:43+00:00] INFO: file[/opt/gitlab/etc/gitlab-rails/env/RAILS_ENV] created file /opt/gitlab/etc/gitlab-rails/env/RAILS_ENV
3277 - create new file /opt/gitlab/etc/gitlab-rails/env/RAILS_ENV[2023-07-20T03:22:43+00:00] INFO: file[/opt/gitlab/etc/gitlab-rails/env/RAILS_ENV] updated file contents /opt/gitlab/etc/gitlab-rails/env/RAILS_ENV
3278 - update content in file /opt/gitlab/etc/gitlab-rails/env/RAILS_ENV from none to ab8e18
3279 - suppressed sensitive resource
3280 * file[/opt/gitlab/etc/gitlab-rails/env/BUNDLE_GEMFILE] action create[2023-07-20T03:22:43+00:00] INFO: file[/opt/gitlab/etc/gitlab-rails/env/BUNDLE_GEMFILE] created file /opt/gitlab/etc/gitlab-rails/env/BUNDLE_GEMFILE
3281 - create new file /opt/gitlab/etc/gitlab-rails/env/BUNDLE_GEMFILE[2023-07-20T03:22:43+00:00] INFO: file[/opt/gitlab/etc/gitlab-rails/env/BUNDLE_GEMFILE] updated file contents /opt/gitlab/etc/gitlab-rails/env/BUNDLE_GEMFILE
3282 - update content in file /opt/gitlab/etc/gitlab-rails/env/BUNDLE_GEMFILE from none to 28d586
3283 - suppressed sensitive resource
3284 * file[/opt/gitlab/etc/gitlab-rails/env/PUMA_WORKER_MAX_MEMORY] action create[2023-07-20T03:22:43+00:00] INFO: file[/opt/gitlab/etc/gitlab-rails/env/PUMA_WORKER_MAX_MEMORY] created file /opt/gitlab/etc/gitlab-rails/env/PUMA_WORKER_MAX_MEMORY
3285 - create new file /opt/gitlab/etc/gitlab-rails/env/PUMA_WORKER_MAX_MEMORY[2023-07-20T03:22:43+00:00] INFO: file[/opt/gitlab/etc/gitlab-rails/env/PUMA_WORKER_MAX_MEMORY] updated file contents /opt/gitlab/etc/gitlab-rails/env/PUMA_WORKER_MAX_MEMORY
3286 - update content in file /opt/gitlab/etc/gitlab-rails/env/PUMA_WORKER_MAX_MEMORY from none to e3b0c4
3287 - suppressed sensitive resource
3288 * file[/opt/gitlab/etc/gitlab-rails/env/SIDEKIQ_MEMORY_KILLER_MAX_RSS] action create[2023-07-20T03:22:43+00:00] INFO: file[/opt/gitlab/etc/gitlab-rails/env/SIDEKIQ_MEMORY_KILLER_MAX_RSS] created file /opt/gitlab/etc/gitlab-rails/env/SIDEKIQ_MEMORY_KILLER_MAX_RSS
3289 - create new file /opt/gitlab/etc/gitlab-rails/env/SIDEKIQ_MEMORY_KILLER_MAX_RSS[2023-07-20T03:22:43+00:00] INFO: file[/opt/gitlab/etc/gitlab-rails/env/SIDEKIQ_MEMORY_KILLER_MAX_RSS] updated file contents /opt/gitlab/etc/gitlab-rails/env/SIDEKIQ_MEMORY_KILLER_MAX_RSS
3290 - update content in file /opt/gitlab/etc/gitlab-rails/env/SIDEKIQ_MEMORY_KILLER_MAX_RSS from none to dd80d7
3291 - suppressed sensitive resource
3292 * file[/opt/gitlab/etc/gitlab-rails/env/PATH] action create[2023-07-20T03:22:43+00:00] INFO: file[/opt/gitlab/etc/gitlab-rails/env/PATH] created file /opt/gitlab/etc/gitlab-rails/env/PATH
3293 - create new file /opt/gitlab/etc/gitlab-rails/env/PATH[2023-07-20T03:22:43+00:00] INFO: file[/opt/gitlab/etc/gitlab-rails/env/PATH] updated file contents /opt/gitlab/etc/gitlab-rails/env/PATH
3294 - update content in file /opt/gitlab/etc/gitlab-rails/env/PATH from none to d5dc07
3295 - suppressed sensitive resource
3296 * file[/opt/gitlab/etc/gitlab-rails/env/ICU_DATA] action create[2023-07-20T03:22:43+00:00] INFO: file[/opt/gitlab/etc/gitlab-rails/env/ICU_DATA] created file /opt/gitlab/etc/gitlab-rails/env/ICU_DATA
3297 - create new file /opt/gitlab/etc/gitlab-rails/env/ICU_DATA[2023-07-20T03:22:43+00:00] INFO: file[/opt/gitlab/etc/gitlab-rails/env/ICU_DATA] updated file contents /opt/gitlab/etc/gitlab-rails/env/ICU_DATA
3298 - update content in file /opt/gitlab/etc/gitlab-rails/env/ICU_DATA from none to a04260
3299 - suppressed sensitive resource
3300 * file[/opt/gitlab/etc/gitlab-rails/env/PYTHONPATH] action create[2023-07-20T03:22:43+00:00] INFO: file[/opt/gitlab/etc/gitlab-rails/env/PYTHONPATH] created file /opt/gitlab/etc/gitlab-rails/env/PYTHONPATH
3301 - create new file /opt/gitlab/etc/gitlab-rails/env/PYTHONPATH[2023-07-20T03:22:43+00:00] INFO: file[/opt/gitlab/etc/gitlab-rails/env/PYTHONPATH] updated file contents /opt/gitlab/etc/gitlab-rails/env/PYTHONPATH
3302 - update content in file /opt/gitlab/etc/gitlab-rails/env/PYTHONPATH from none to 98c9f4
3303 - suppressed sensitive resource
3304 * file[/opt/gitlab/etc/gitlab-rails/env/EXECJS_RUNTIME] action create[2023-07-20T03:22:43+00:00] INFO: file[/opt/gitlab/etc/gitlab-rails/env/EXECJS_RUNTIME] created file /opt/gitlab/etc/gitlab-rails/env/EXECJS_RUNTIME
3305 - create new file /opt/gitlab/etc/gitlab-rails/env/EXECJS_RUNTIME[2023-07-20T03:22:43+00:00] INFO: file[/opt/gitlab/etc/gitlab-rails/env/EXECJS_RUNTIME] updated file contents /opt/gitlab/etc/gitlab-rails/env/EXECJS_RUNTIME
3306 - update content in file /opt/gitlab/etc/gitlab-rails/env/EXECJS_RUNTIME from none to 75081b
3307 - suppressed sensitive resource
3308 * file[/opt/gitlab/etc/gitlab-rails/env/TZ] action create[2023-07-20T03:22:43+00:00] INFO: file[/opt/gitlab/etc/gitlab-rails/env/TZ] created file /opt/gitlab/etc/gitlab-rails/env/TZ
3309 - create new file /opt/gitlab/etc/gitlab-rails/env/TZ[2023-07-20T03:22:43+00:00] INFO: file[/opt/gitlab/etc/gitlab-rails/env/TZ] updated file contents /opt/gitlab/etc/gitlab-rails/env/TZ
3310 - update content in file /opt/gitlab/etc/gitlab-rails/env/TZ from none to 983a95
3311 - suppressed sensitive resource
3312 * file[/opt/gitlab/etc/gitlab-rails/env/SSL_CERT_DIR] action create[2023-07-20T03:22:43+00:00] INFO: file[/opt/gitlab/etc/gitlab-rails/env/SSL_CERT_DIR] created file /opt/gitlab/etc/gitlab-rails/env/SSL_CERT_DIR
3313 - create new file /opt/gitlab/etc/gitlab-rails/env/SSL_CERT_DIR[2023-07-20T03:22:43+00:00] INFO: file[/opt/gitlab/etc/gitlab-rails/env/SSL_CERT_DIR] updated file contents /opt/gitlab/etc/gitlab-rails/env/SSL_CERT_DIR
3314 - update content in file /opt/gitlab/etc/gitlab-rails/env/SSL_CERT_DIR from none to 4f45cf
3315 - suppressed sensitive resource
3316 * file[/opt/gitlab/etc/gitlab-rails/env/SSL_CERT_FILE] action create[2023-07-20T03:22:43+00:00] INFO: file[/opt/gitlab/etc/gitlab-rails/env/SSL_CERT_FILE] created file /opt/gitlab/etc/gitlab-rails/env/SSL_CERT_FILE
3317 - create new file /opt/gitlab/etc/gitlab-rails/env/SSL_CERT_FILE[2023-07-20T03:22:43+00:00] INFO: file[/opt/gitlab/etc/gitlab-rails/env/SSL_CERT_FILE] updated file contents /opt/gitlab/etc/gitlab-rails/env/SSL_CERT_FILE
3318 - update content in file /opt/gitlab/etc/gitlab-rails/env/SSL_CERT_FILE from none to f7ce84
3319 - suppressed sensitive resource
3320
3321 * link[/opt/gitlab/embedded/service/gitlab-rails/tmp] action create[2023-07-20T03:22:43+00:00] INFO: link[/opt/gitlab/embedded/service/gitlab-rails/tmp] created
3322 - create symlink at /opt/gitlab/embedded/service/gitlab-rails/tmp to /var/opt/gitlab/gitlab-rails/tmp
3323 * link[/opt/gitlab/embedded/service/gitlab-rails/public/uploads] action create[2023-07-20T03:22:43+00:00] INFO: link[/opt/gitlab/embedded/service/gitlab-rails/public/uploads] created
3324 - create symlink at /opt/gitlab/embedded/service/gitlab-rails/public/uploads to /var/opt/gitlab/gitlab-rails/uploads
3325 * link[/opt/gitlab/embedded/service/gitlab-rails/log] action create[2023-07-20T03:22:43+00:00] INFO: link[/opt/gitlab/embedded/service/gitlab-rails/log] created
3326 - create symlink at /opt/gitlab/embedded/service/gitlab-rails/log to /var/log/gitlab/gitlab-rails
3327 * link[/var/log/gitlab/gitlab-rails/sidekiq.log] action delete (skipped due to only_if)
3328 * file[/opt/gitlab/embedded/service/gitlab-rails/db/structure.sql] action create[2023-07-20T03:22:43+00:00] INFO: file[/opt/gitlab/embedded/service/gitlab-rails/db/structure.sql] owner changed to 998
3329 - change owner from 'root' to 'git'
3330 * remote_file[/var/opt/gitlab/gitlab-rails/VERSION] action create[2023-07-20T03:22:43+00:00] INFO: remote_file[/var/opt/gitlab/gitlab-rails/VERSION] created file /var/opt/gitlab/gitlab-rails/VERSION
3331 - create new file /var/opt/gitlab/gitlab-rails/VERSION[2023-07-20T03:22:43+00:00] INFO: remote_file[/var/opt/gitlab/gitlab-rails/VERSION] updated file contents /var/opt/gitlab/gitlab-rails/VERSION
3332 - update content in file /var/opt/gitlab/gitlab-rails/VERSION from none to 5dc40e
3333 --- /var/opt/gitlab/gitlab-rails/VERSION 2023-07-20 03:22:43.000000000 +0000
3334 +++ /var/opt/gitlab/gitlab-rails/.chef-VERSION20230720-37-3c2tpx 2023-07-20 03:22:43.000000000 +0000
3335 @@ -1 +1,2 @@
3336 +16.2.0-pre
3337 * remote_file[/var/opt/gitlab/gitlab-rails/REVISION] action create[2023-07-20T03:22:43+00:00] INFO: remote_file[/var/opt/gitlab/gitlab-rails/REVISION] created file /var/opt/gitlab/gitlab-rails/REVISION
3338 - create new file /var/opt/gitlab/gitlab-rails/REVISION[2023-07-20T03:22:43+00:00] INFO: remote_file[/var/opt/gitlab/gitlab-rails/REVISION] updated file contents /var/opt/gitlab/gitlab-rails/REVISION
3339 - update content in file /var/opt/gitlab/gitlab-rails/REVISION from none to 23111e
3340 --- /var/opt/gitlab/gitlab-rails/REVISION 2023-07-20 03:22:43.000000000 +0000
3341 +++ /var/opt/gitlab/gitlab-rails/.chef-REVISION20230720-37-z7qr8t 2023-07-20 03:22:43.000000000 +0000
3342 @@ -1 +1,2 @@
3343 +3913740266a
3344 * version_file[Create version file for Rails] action create
3345 * file[/var/opt/gitlab/gitlab-rails/RUBY_VERSION] action create[2023-07-20T03:22:43+00:00] INFO: file[/var/opt/gitlab/gitlab-rails/RUBY_VERSION] created file /var/opt/gitlab/gitlab-rails/RUBY_VERSION
3346 - create new file /var/opt/gitlab/gitlab-rails/RUBY_VERSION[2023-07-20T03:22:43+00:00] INFO: file[/var/opt/gitlab/gitlab-rails/RUBY_VERSION] updated file contents /var/opt/gitlab/gitlab-rails/RUBY_VERSION
3347 - update content in file /var/opt/gitlab/gitlab-rails/RUBY_VERSION from none to b4a021
3348 --- /var/opt/gitlab/gitlab-rails/RUBY_VERSION 2023-07-20 03:22:43.000000000 +0000
3349 +++ /var/opt/gitlab/gitlab-rails/.chef-RUBY_VERSION20230720-37-tpmyid 2023-07-20 03:22:43.000000000 +0000
3350 @@ -1 +1,2 @@
3351 +ruby 3.0.6p216 (2023-03-30 revision 23a532679b) [x86_64-linux]
3352
3353 * execute[clear the gitlab-rails cache] action nothing (skipped due to action :nothing)
3354 * file[/var/opt/gitlab/gitlab-rails/config.ru] action delete (up to date)
3355Recipe: gitlab::selinux
3356 * bash[Set proper security context on ssh files for selinux] action nothing (skipped due to action :nothing)
3357Recipe: gitlab::add_trusted_certs
3358 * directory[/etc/gitlab/trusted-certs] action create[2023-07-20T03:22:43+00:00] INFO: directory[/etc/gitlab/trusted-certs] created directory /etc/gitlab/trusted-certs
3359 - create new directory /etc/gitlab/trusted-certs[2023-07-20T03:22:43+00:00] INFO: directory[/etc/gitlab/trusted-certs] mode changed to 755
3360 - change mode from '' to '0755'
3361 * directory[/opt/gitlab/embedded/ssl/certs] action create (up to date)
3362 * file[/opt/gitlab/embedded/ssl/certs/README] action create[2023-07-20T03:22:43+00:00] INFO: file[/opt/gitlab/embedded/ssl/certs/README] created file /opt/gitlab/embedded/ssl/certs/README
3363 - create new file /opt/gitlab/embedded/ssl/certs/README[2023-07-20T03:22:43+00:00] INFO: file[/opt/gitlab/embedded/ssl/certs/README] updated file contents /opt/gitlab/embedded/ssl/certs/README
3364 - update content in file /opt/gitlab/embedded/ssl/certs/README from none to 623059
3365 --- /opt/gitlab/embedded/ssl/certs/README 2023-07-20 03:22:43.000000000 +0000
3366 +++ /opt/gitlab/embedded/ssl/certs/.chef-README20230720-37-c7kvz3 2023-07-20 03:22:43.000000000 +0000
3367 @@ -1,3 +1,6 @@
3368 +This directory is managed by omnibus-gitlab.
3369 + Any file placed in this directory will be ignored
3370 +. Place certificates in /etc/gitlab/trusted-certs.[2023-07-20T03:22:43+00:00] INFO: file[/opt/gitlab/embedded/ssl/certs/README] mode changed to 644
3371 - change mode from '' to '0644'
3372 * ruby_block[Move existing certs and link to /opt/gitlab/embedded/ssl/certs] action run
3373 * Moving existing certificates found in /opt/gitlab/embedded/ssl/certs
3374 * Symlinking existing certificates found in /etc/gitlab/trusted-certs
3375[2023-07-20T03:22:43+00:00] INFO: ruby_block[Move existing certs and link to /opt/gitlab/embedded/ssl/certs] called
3376 - execute the ruby block Move existing certs and link to /opt/gitlab/embedded/ssl/certs
3377Recipe: gitlab::default
3378 * service[create a temporary puma service] action nothing (skipped due to action :nothing)
3379 * service[create a temporary sidekiq service] action nothing (skipped due to action :nothing)
3380 * service[create a temporary mailroom service] action nothing (skipped due to action :nothing)
3381Recipe: gitlab::gitlab-shell
3382 * storage_directory[/var/opt/gitlab/.ssh] action create
3383 * ruby_block[directory resource: /var/opt/gitlab/.ssh] action run[2023-07-20T03:22:44+00:00] INFO: ruby_block[directory resource: /var/opt/gitlab/.ssh] called
3384 - execute the ruby block directory resource: /var/opt/gitlab/.ssh
3385
3386 * directory[/var/opt/gitlab/gitlab-shell] action create[2023-07-20T03:22:44+00:00] INFO: directory[/var/opt/gitlab/gitlab-shell] created directory /var/opt/gitlab/gitlab-shell
3387 - create new directory /var/opt/gitlab/gitlab-shell[2023-07-20T03:22:44+00:00] INFO: directory[/var/opt/gitlab/gitlab-shell] owner changed to 998
3388[2023-07-20T03:22:44+00:00] INFO: directory[/var/opt/gitlab/gitlab-shell] mode changed to 700
3389 - change mode from '' to '0700'
3390 - change owner from '' to 'git'
3391 * directory[/var/log/gitlab/gitlab-shell] action create[2023-07-20T03:22:44+00:00] INFO: directory[/var/log/gitlab/gitlab-shell] created directory /var/log/gitlab/gitlab-shell
3392 - create new directory /var/log/gitlab/gitlab-shell[2023-07-20T03:22:44+00:00] INFO: directory[/var/log/gitlab/gitlab-shell] owner changed to 998
3393[2023-07-20T03:22:44+00:00] INFO: directory[/var/log/gitlab/gitlab-shell] mode changed to 700
3394 - change mode from '' to '0700'
3395 - change owner from '' to 'git'
3396 * bash[generate gitlab-sshd host keys] action run (skipped due to only_if)
3397 * templatesymlink[Create a config.yml and create a symlink to Rails root] action create
3398 * template[/var/opt/gitlab/gitlab-shell/config.yml] action create[2023-07-20T03:22:44+00:00] INFO: template[/var/opt/gitlab/gitlab-shell/config.yml] created file /var/opt/gitlab/gitlab-shell/config.yml
3399 - create new file /var/opt/gitlab/gitlab-shell/config.yml[2023-07-20T03:22:44+00:00] INFO: template[/var/opt/gitlab/gitlab-shell/config.yml] updated file contents /var/opt/gitlab/gitlab-shell/config.yml
3400 - update content in file /var/opt/gitlab/gitlab-shell/config.yml from none to a5b7b3
3401 - suppressed sensitive resource[2023-07-20T03:22:44+00:00] INFO: template[/var/opt/gitlab/gitlab-shell/config.yml] owner changed to 0
3402[2023-07-20T03:22:44+00:00] INFO: template[/var/opt/gitlab/gitlab-shell/config.yml] group changed to 998
3403[2023-07-20T03:22:44+00:00] INFO: template[/var/opt/gitlab/gitlab-shell/config.yml] mode changed to 640
3404 - change mode from '' to '0640'
3405 - change owner from '' to 'root'
3406 - change group from '' to 'git'
3407 * link[Link /opt/gitlab/embedded/service/gitlab-shell/config.yml to /var/opt/gitlab/gitlab-shell/config.yml] action create[2023-07-20T03:22:44+00:00] INFO: link[Link /opt/gitlab/embedded/service/gitlab-shell/config.yml to /var/opt/gitlab/gitlab-shell/config.yml] created
3408 - create symlink at /opt/gitlab/embedded/service/gitlab-shell/config.yml to /var/opt/gitlab/gitlab-shell/config.yml
3409
3410 * link[/opt/gitlab/embedded/service/gitlab-shell/.gitlab_shell_secret] action create[2023-07-20T03:22:44+00:00] INFO: link[/opt/gitlab/embedded/service/gitlab-shell/.gitlab_shell_secret] created
3411 - create symlink at /opt/gitlab/embedded/service/gitlab-shell/.gitlab_shell_secret to /opt/gitlab/embedded/service/gitlab-rails/.gitlab_shell_secret
3412 * file[/var/opt/gitlab/.ssh/authorized_keys] action create_if_missing[2023-07-20T03:22:44+00:00] INFO: file[/var/opt/gitlab/.ssh/authorized_keys] created file /var/opt/gitlab/.ssh/authorized_keys
3413 - create new file /var/opt/gitlab/.ssh/authorized_keys[2023-07-20T03:22:44+00:00] INFO: file[/var/opt/gitlab/.ssh/authorized_keys] owner changed to 998
3414[2023-07-20T03:22:44+00:00] INFO: file[/var/opt/gitlab/.ssh/authorized_keys] group changed to 998
3415[2023-07-20T03:22:44+00:00] INFO: file[/var/opt/gitlab/.ssh/authorized_keys] mode changed to 600
3416 - change mode from '' to '0600'
3417 - change owner from '' to 'git'
3418 - change group from '' to 'git'
3419 * env_dir[/opt/gitlab/etc/gitlab-sshd/env] action nothing (skipped due to action :nothing)
3420 * service[gitlab-sshd] action nothing (skipped due to action :nothing)
3421 * runit_service[gitlab-sshd] action disable
3422 * ruby_block[disable gitlab-sshd] action run (skipped due to only_if)
3423 (up to date)
3424Recipe: package::sysctl
3425 * execute[reload all sysctl conf] action nothing (skipped due to action :nothing)
3426Recipe: logrotate::folders_and_configs
3427 * directory[/var/opt/gitlab/logrotate] action create[2023-07-20T03:22:44+00:00] INFO: directory[/var/opt/gitlab/logrotate] created directory /var/opt/gitlab/logrotate
3428 - create new directory /var/opt/gitlab/logrotate[2023-07-20T03:22:44+00:00] INFO: directory[/var/opt/gitlab/logrotate] mode changed to 700
3429 - change mode from '' to '0700'
3430 * directory[/var/opt/gitlab/logrotate/logrotate.d] action create[2023-07-20T03:22:44+00:00] INFO: directory[/var/opt/gitlab/logrotate/logrotate.d] created directory /var/opt/gitlab/logrotate/logrotate.d
3431 - create new directory /var/opt/gitlab/logrotate/logrotate.d[2023-07-20T03:22:44+00:00] INFO: directory[/var/opt/gitlab/logrotate/logrotate.d] mode changed to 700
3432 - change mode from '' to '0700'
3433 * directory[/var/log/gitlab/logrotate] action create[2023-07-20T03:22:44+00:00] INFO: directory[/var/log/gitlab/logrotate] created directory /var/log/gitlab/logrotate
3434 - create new directory /var/log/gitlab/logrotate[2023-07-20T03:22:44+00:00] INFO: directory[/var/log/gitlab/logrotate] owner changed to 0
3435[2023-07-20T03:22:44+00:00] INFO: directory[/var/log/gitlab/logrotate] mode changed to 700
3436 - change mode from '' to '0700'
3437 - change owner from '' to 'root'
3438 * template[/var/opt/gitlab/logrotate/logrotate.conf] action create[2023-07-20T03:22:44+00:00] INFO: template[/var/opt/gitlab/logrotate/logrotate.conf] created file /var/opt/gitlab/logrotate/logrotate.conf
3439 - create new file /var/opt/gitlab/logrotate/logrotate.conf[2023-07-20T03:22:44+00:00] INFO: template[/var/opt/gitlab/logrotate/logrotate.conf] updated file contents /var/opt/gitlab/logrotate/logrotate.conf
3440 - update content in file /var/opt/gitlab/logrotate/logrotate.conf from none to c1c320
3441 --- /var/opt/gitlab/logrotate/logrotate.conf 2023-07-20 03:22:44.000000000 +0000
3442 +++ /var/opt/gitlab/logrotate/.chef-logrotate20230720-37-o1tyir.conf 2023-07-20 03:22:44.000000000 +0000
3443 @@ -1,12 +1,24 @@
3444 +# Generated by 'gitlab-ctl reconfigure'.
3445 +# Modifications will be overwritten!
3446 +
3447 +include /var/opt/gitlab/logrotate/logrotate.d/nginx
3448 +include /var/opt/gitlab/logrotate/logrotate.d/puma
3449 +include /var/opt/gitlab/logrotate/logrotate.d/gitlab-rails
3450 +include /var/opt/gitlab/logrotate/logrotate.d/gitlab-shell
3451 +include /var/opt/gitlab/logrotate/logrotate.d/gitlab-workhorse
3452 +include /var/opt/gitlab/logrotate/logrotate.d/gitlab-pages
3453 +include /var/opt/gitlab/logrotate/logrotate.d/gitlab-kas
3454 +include /var/opt/gitlab/logrotate/logrotate.d/gitaly
3455 +include /var/opt/gitlab/logrotate/logrotate.d/mailroom[2023-07-20T03:22:44+00:00] INFO: template[/var/opt/gitlab/logrotate/logrotate.conf] mode changed to 644
3456 - change mode from '' to '0644'
3457 * template[/var/opt/gitlab/logrotate/logrotate.d/nginx] action create[2023-07-20T03:22:44+00:00] INFO: template[/var/opt/gitlab/logrotate/logrotate.d/nginx] created file /var/opt/gitlab/logrotate/logrotate.d/nginx
3458 - create new file /var/opt/gitlab/logrotate/logrotate.d/nginx[2023-07-20T03:22:44+00:00] INFO: template[/var/opt/gitlab/logrotate/logrotate.d/nginx] updated file contents /var/opt/gitlab/logrotate/logrotate.d/nginx
3459 - update content in file /var/opt/gitlab/logrotate/logrotate.d/nginx from none to e17e19
3460 --- /var/opt/gitlab/logrotate/logrotate.d/nginx 2023-07-20 03:22:44.000000000 +0000
3461 +++ /var/opt/gitlab/logrotate/logrotate.d/.chef-nginx20230720-37-g4gb34 2023-07-20 03:22:44.000000000 +0000
3462 @@ -1,17 +1,34 @@
3463 +# Generated by gitlab-ctl reconfigure
3464 +# Modifications will be overwritten!
3465 +
3466 +/var/log/gitlab/nginx/*.log {
3467 + su root root
3468 + daily
3469 +
3470 +
3471 + rotate 30
3472 + compress
3473 + copytruncate
3474 + missingok
3475 + notifempty
3476 + postrotate
3477 +
3478 + endscript
3479 +}
3480 * template[/var/opt/gitlab/logrotate/logrotate.d/puma] action create[2023-07-20T03:22:44+00:00] INFO: template[/var/opt/gitlab/logrotate/logrotate.d/puma] created file /var/opt/gitlab/logrotate/logrotate.d/puma
3481 - create new file /var/opt/gitlab/logrotate/logrotate.d/puma[2023-07-20T03:22:44+00:00] INFO: template[/var/opt/gitlab/logrotate/logrotate.d/puma] updated file contents /var/opt/gitlab/logrotate/logrotate.d/puma
3482 - update content in file /var/opt/gitlab/logrotate/logrotate.d/puma from none to 44ad27
3483 --- /var/opt/gitlab/logrotate/logrotate.d/puma 2023-07-20 03:22:44.000000000 +0000
3484 +++ /var/opt/gitlab/logrotate/logrotate.d/.chef-puma20230720-37-lnr509 2023-07-20 03:22:44.000000000 +0000
3485 @@ -1,17 +1,34 @@
3486 +# Generated by gitlab-ctl reconfigure
3487 +# Modifications will be overwritten!
3488 +
3489 +/var/log/gitlab/puma/*.log {
3490 + su git git
3491 + daily
3492 +
3493 +
3494 + rotate 30
3495 + compress
3496 + copytruncate
3497 + missingok
3498 + notifempty
3499 + postrotate
3500 +
3501 + endscript
3502 +}
3503 * template[/var/opt/gitlab/logrotate/logrotate.d/gitlab-rails] action create[2023-07-20T03:22:44+00:00] INFO: template[/var/opt/gitlab/logrotate/logrotate.d/gitlab-rails] created file /var/opt/gitlab/logrotate/logrotate.d/gitlab-rails
3504 - create new file /var/opt/gitlab/logrotate/logrotate.d/gitlab-rails[2023-07-20T03:22:44+00:00] INFO: template[/var/opt/gitlab/logrotate/logrotate.d/gitlab-rails] updated file contents /var/opt/gitlab/logrotate/logrotate.d/gitlab-rails
3505 - update content in file /var/opt/gitlab/logrotate/logrotate.d/gitlab-rails from none to 26254a
3506 --- /var/opt/gitlab/logrotate/logrotate.d/gitlab-rails 2023-07-20 03:22:44.000000000 +0000
3507 +++ /var/opt/gitlab/logrotate/logrotate.d/.chef-gitlab-rails20230720-37-954z0z 2023-07-20 03:22:44.000000000 +0000
3508 @@ -1,17 +1,34 @@
3509 +# Generated by gitlab-ctl reconfigure
3510 +# Modifications will be overwritten!
3511 +
3512 +/var/log/gitlab/gitlab-rails/*.log {
3513 + su git git
3514 + daily
3515 +
3516 +
3517 + rotate 30
3518 + compress
3519 + copytruncate
3520 + missingok
3521 + notifempty
3522 + postrotate
3523 +
3524 + endscript
3525 +}
3526 * template[/var/opt/gitlab/logrotate/logrotate.d/gitlab-shell] action create[2023-07-20T03:22:44+00:00] INFO: template[/var/opt/gitlab/logrotate/logrotate.d/gitlab-shell] created file /var/opt/gitlab/logrotate/logrotate.d/gitlab-shell
3527 - create new file /var/opt/gitlab/logrotate/logrotate.d/gitlab-shell[2023-07-20T03:22:44+00:00] INFO: template[/var/opt/gitlab/logrotate/logrotate.d/gitlab-shell] updated file contents /var/opt/gitlab/logrotate/logrotate.d/gitlab-shell
3528 - update content in file /var/opt/gitlab/logrotate/logrotate.d/gitlab-shell from none to 859f56
3529 --- /var/opt/gitlab/logrotate/logrotate.d/gitlab-shell 2023-07-20 03:22:44.000000000 +0000
3530 +++ /var/opt/gitlab/logrotate/logrotate.d/.chef-gitlab-shell20230720-37-vcuweg 2023-07-20 03:22:44.000000000 +0000
3531 @@ -1,17 +1,34 @@
3532 +# Generated by gitlab-ctl reconfigure
3533 +# Modifications will be overwritten!
3534 +
3535 +/var/log/gitlab/gitlab-shell/*.log {
3536 + su git git
3537 + daily
3538 +
3539 +
3540 + rotate 30
3541 + compress
3542 + copytruncate
3543 + missingok
3544 + notifempty
3545 + postrotate
3546 +
3547 + endscript
3548 +}
3549 * template[/var/opt/gitlab/logrotate/logrotate.d/gitlab-workhorse] action create[2023-07-20T03:22:44+00:00] INFO: template[/var/opt/gitlab/logrotate/logrotate.d/gitlab-workhorse] created file /var/opt/gitlab/logrotate/logrotate.d/gitlab-workhorse
3550 - create new file /var/opt/gitlab/logrotate/logrotate.d/gitlab-workhorse[2023-07-20T03:22:44+00:00] INFO: template[/var/opt/gitlab/logrotate/logrotate.d/gitlab-workhorse] updated file contents /var/opt/gitlab/logrotate/logrotate.d/gitlab-workhorse
3551 - update content in file /var/opt/gitlab/logrotate/logrotate.d/gitlab-workhorse from none to 3453f6
3552 --- /var/opt/gitlab/logrotate/logrotate.d/gitlab-workhorse 2023-07-20 03:22:44.000000000 +0000
3553 +++ /var/opt/gitlab/logrotate/logrotate.d/.chef-gitlab-workhorse20230720-37-zvsclr 2023-07-20 03:22:44.000000000 +0000
3554 @@ -1,17 +1,34 @@
3555 +# Generated by gitlab-ctl reconfigure
3556 +# Modifications will be overwritten!
3557 +
3558 +/var/log/gitlab/gitlab-workhorse/*.log {
3559 + su git git
3560 + daily
3561 +
3562 +
3563 + rotate 30
3564 + compress
3565 + copytruncate
3566 + missingok
3567 + notifempty
3568 + postrotate
3569 +
3570 + endscript
3571 +}
3572 * template[/var/opt/gitlab/logrotate/logrotate.d/gitlab-pages] action create[2023-07-20T03:22:44+00:00] INFO: template[/var/opt/gitlab/logrotate/logrotate.d/gitlab-pages] created file /var/opt/gitlab/logrotate/logrotate.d/gitlab-pages
3573 - create new file /var/opt/gitlab/logrotate/logrotate.d/gitlab-pages[2023-07-20T03:22:44+00:00] INFO: template[/var/opt/gitlab/logrotate/logrotate.d/gitlab-pages] updated file contents /var/opt/gitlab/logrotate/logrotate.d/gitlab-pages
3574 - update content in file /var/opt/gitlab/logrotate/logrotate.d/gitlab-pages from none to 3c118a
3575 --- /var/opt/gitlab/logrotate/logrotate.d/gitlab-pages 2023-07-20 03:22:44.000000000 +0000
3576 +++ /var/opt/gitlab/logrotate/logrotate.d/.chef-gitlab-pages20230720-37-y5rto5 2023-07-20 03:22:44.000000000 +0000
3577 @@ -1,17 +1,34 @@
3578 +# Generated by gitlab-ctl reconfigure
3579 +# Modifications will be overwritten!
3580 +
3581 +/var/log/gitlab/gitlab-pages/*.log {
3582 + su git git
3583 + daily
3584 +
3585 +
3586 + rotate 30
3587 + compress
3588 + copytruncate
3589 + missingok
3590 + notifempty
3591 + postrotate
3592 +
3593 + endscript
3594 +}
3595 * template[/var/opt/gitlab/logrotate/logrotate.d/gitlab-kas] action create[2023-07-20T03:22:44+00:00] INFO: template[/var/opt/gitlab/logrotate/logrotate.d/gitlab-kas] created file /var/opt/gitlab/logrotate/logrotate.d/gitlab-kas
3596 - create new file /var/opt/gitlab/logrotate/logrotate.d/gitlab-kas[2023-07-20T03:22:44+00:00] INFO: template[/var/opt/gitlab/logrotate/logrotate.d/gitlab-kas] updated file contents /var/opt/gitlab/logrotate/logrotate.d/gitlab-kas
3597 - update content in file /var/opt/gitlab/logrotate/logrotate.d/gitlab-kas from none to 0ba4f0
3598 --- /var/opt/gitlab/logrotate/logrotate.d/gitlab-kas 2023-07-20 03:22:44.000000000 +0000
3599 +++ /var/opt/gitlab/logrotate/logrotate.d/.chef-gitlab-kas20230720-37-89ly30 2023-07-20 03:22:44.000000000 +0000
3600 @@ -1,17 +1,34 @@
3601 +# Generated by gitlab-ctl reconfigure
3602 +# Modifications will be overwritten!
3603 +
3604 +/var/log/gitlab/gitlab-kas/*.log {
3605 + su git git
3606 + daily
3607 +
3608 +
3609 + rotate 30
3610 + compress
3611 + copytruncate
3612 + missingok
3613 + notifempty
3614 + postrotate
3615 +
3616 + endscript
3617 +}
3618 * template[/var/opt/gitlab/logrotate/logrotate.d/gitaly] action create[2023-07-20T03:22:44+00:00] INFO: template[/var/opt/gitlab/logrotate/logrotate.d/gitaly] created file /var/opt/gitlab/logrotate/logrotate.d/gitaly
3619 - create new file /var/opt/gitlab/logrotate/logrotate.d/gitaly[2023-07-20T03:22:44+00:00] INFO: template[/var/opt/gitlab/logrotate/logrotate.d/gitaly] updated file contents /var/opt/gitlab/logrotate/logrotate.d/gitaly
3620 - update content in file /var/opt/gitlab/logrotate/logrotate.d/gitaly from none to 02bda7
3621 --- /var/opt/gitlab/logrotate/logrotate.d/gitaly 2023-07-20 03:22:44.000000000 +0000
3622 +++ /var/opt/gitlab/logrotate/logrotate.d/.chef-gitaly20230720-37-bvj6yw 2023-07-20 03:22:44.000000000 +0000
3623 @@ -1,17 +1,34 @@
3624 +# Generated by gitlab-ctl reconfigure
3625 +# Modifications will be overwritten!
3626 +
3627 +/var/log/gitlab/gitaly/*.log {
3628 + su git git
3629 + daily
3630 +
3631 +
3632 + rotate 30
3633 + compress
3634 + copytruncate
3635 + missingok
3636 + notifempty
3637 + postrotate
3638 +
3639 + endscript
3640 +}
3641 * template[/var/opt/gitlab/logrotate/logrotate.d/mailroom] action create[2023-07-20T03:22:44+00:00] INFO: template[/var/opt/gitlab/logrotate/logrotate.d/mailroom] created file /var/opt/gitlab/logrotate/logrotate.d/mailroom
3642 - create new file /var/opt/gitlab/logrotate/logrotate.d/mailroom[2023-07-20T03:22:44+00:00] INFO: template[/var/opt/gitlab/logrotate/logrotate.d/mailroom] updated file contents /var/opt/gitlab/logrotate/logrotate.d/mailroom
3643 - update content in file /var/opt/gitlab/logrotate/logrotate.d/mailroom from none to 8fcea3
3644 --- /var/opt/gitlab/logrotate/logrotate.d/mailroom 2023-07-20 03:22:44.000000000 +0000
3645 +++ /var/opt/gitlab/logrotate/logrotate.d/.chef-mailroom20230720-37-uf2d0h 2023-07-20 03:22:44.000000000 +0000
3646 @@ -1,17 +1,34 @@
3647 +# Generated by gitlab-ctl reconfigure
3648 +# Modifications will be overwritten!
3649 +
3650 +/var/log/gitlab/mailroom/*.log {
3651 + su git git
3652 + daily
3653 +
3654 +
3655 + rotate 30
3656 + compress
3657 + copytruncate
3658 + missingok
3659 + notifempty
3660 + postrotate
3661 +
3662 + endscript
3663 +}
3664Recipe: logrotate::enable
3665 * service[logrotate] action nothing (skipped due to action :nothing)
3666 * runit_service[logrotate] action enable
3667 * ruby_block[restart_service] action nothing (skipped due to action :nothing)
3668 * ruby_block[restart_log_service] action nothing (skipped due to action :nothing)
3669 * ruby_block[reload_log_service] action nothing (skipped due to action :nothing)
3670 * directory[/opt/gitlab/sv/logrotate] action create[2023-07-20T03:22:44+00:00] INFO: directory[/opt/gitlab/sv/logrotate] created directory /opt/gitlab/sv/logrotate
3671 - create new directory /opt/gitlab/sv/logrotate[2023-07-20T03:22:44+00:00] INFO: directory[/opt/gitlab/sv/logrotate] owner changed to 0
3672[2023-07-20T03:22:44+00:00] INFO: directory[/opt/gitlab/sv/logrotate] group changed to 0
3673[2023-07-20T03:22:44+00:00] INFO: directory[/opt/gitlab/sv/logrotate] mode changed to 755
3674 - change mode from '' to '0755'
3675 - change owner from '' to 'root'
3676 - change group from '' to 'root'
3677 * template[/opt/gitlab/sv/logrotate/run] action create[2023-07-20T03:22:44+00:00] INFO: template[/opt/gitlab/sv/logrotate/run] created file /opt/gitlab/sv/logrotate/run
3678 - create new file /opt/gitlab/sv/logrotate/run[2023-07-20T03:22:44+00:00] INFO: template[/opt/gitlab/sv/logrotate/run] updated file contents /opt/gitlab/sv/logrotate/run
3679 - update content in file /opt/gitlab/sv/logrotate/run from none to 07f1b6
3680 --- /opt/gitlab/sv/logrotate/run 2023-07-20 03:22:44.000000000 +0000
3681 +++ /opt/gitlab/sv/logrotate/.chef-run20230720-37-xiyysv 2023-07-20 03:22:44.000000000 +0000
3682 @@ -1,10 +1,20 @@
3683 +#!/bin/sh
3684 +exec 2>&1
3685 +
3686 +cd /var/opt/gitlab/logrotate
3687 +
3688 +exec /opt/gitlab/embedded/bin/chpst -P /usr/bin/env \
3689 + dir=/var/opt/gitlab/logrotate \
3690 + pre_sleep=600 \
3691 + post_sleep=3000 \
3692 + /opt/gitlab/embedded/bin/gitlab-logrotate-wrapper[2023-07-20T03:22:44+00:00] INFO: template[/opt/gitlab/sv/logrotate/run] owner changed to 0
3693[2023-07-20T03:22:44+00:00] INFO: template[/opt/gitlab/sv/logrotate/run] group changed to 0
3694[2023-07-20T03:22:44+00:00] INFO: template[/opt/gitlab/sv/logrotate/run] mode changed to 755
3695 - change mode from '' to '0755'
3696 - change owner from '' to 'root'
3697 - change group from '' to 'root'
3698 * directory[/opt/gitlab/sv/logrotate/log] action create[2023-07-20T03:22:44+00:00] INFO: directory[/opt/gitlab/sv/logrotate/log] created directory /opt/gitlab/sv/logrotate/log
3699 - create new directory /opt/gitlab/sv/logrotate/log[2023-07-20T03:22:44+00:00] INFO: directory[/opt/gitlab/sv/logrotate/log] owner changed to 0
3700[2023-07-20T03:22:44+00:00] INFO: directory[/opt/gitlab/sv/logrotate/log] group changed to 0
3701[2023-07-20T03:22:44+00:00] INFO: directory[/opt/gitlab/sv/logrotate/log] mode changed to 755
3702 - change mode from '' to '0755'
3703 - change owner from '' to 'root'
3704 - change group from '' to 'root'
3705 * directory[/opt/gitlab/sv/logrotate/log/main] action create[2023-07-20T03:22:44+00:00] INFO: directory[/opt/gitlab/sv/logrotate/log/main] created directory /opt/gitlab/sv/logrotate/log/main
3706 - create new directory /opt/gitlab/sv/logrotate/log/main[2023-07-20T03:22:44+00:00] INFO: directory[/opt/gitlab/sv/logrotate/log/main] owner changed to 0
3707[2023-07-20T03:22:44+00:00] INFO: directory[/opt/gitlab/sv/logrotate/log/main] group changed to 0
3708[2023-07-20T03:22:44+00:00] INFO: directory[/opt/gitlab/sv/logrotate/log/main] mode changed to 755
3709 - change mode from '' to '0755'
3710 - change owner from '' to 'root'
3711 - change group from '' to 'root'
3712 * template[/opt/gitlab/sv/logrotate/log/config] action create[2023-07-20T03:22:44+00:00] INFO: template[/opt/gitlab/sv/logrotate/log/config] created file /opt/gitlab/sv/logrotate/log/config
3713 - create new file /opt/gitlab/sv/logrotate/log/config[2023-07-20T03:22:44+00:00] INFO: template[/opt/gitlab/sv/logrotate/log/config] updated file contents /opt/gitlab/sv/logrotate/log/config
3714 - update content in file /opt/gitlab/sv/logrotate/log/config from none to 623c00
3715 --- /opt/gitlab/sv/logrotate/log/config 2023-07-20 03:22:44.000000000 +0000
3716 +++ /opt/gitlab/sv/logrotate/log/.chef-config20230720-37-sfi7vv 2023-07-20 03:22:44.000000000 +0000
3717 @@ -1,6 +1,12 @@
3718 +s209715200
3719 +n30
3720 +t86400
3721 +!gzip
3722 +
3723 +[2023-07-20T03:22:44+00:00] INFO: template[/opt/gitlab/sv/logrotate/log/config] owner changed to 0
3724[2023-07-20T03:22:44+00:00] INFO: template[/opt/gitlab/sv/logrotate/log/config] group changed to 0
3725[2023-07-20T03:22:44+00:00] INFO: template[/opt/gitlab/sv/logrotate/log/config] mode changed to 644
3726 - change mode from '' to '0644'
3727 - change owner from '' to 'root'
3728 - change group from '' to 'root'
3729[2023-07-20T03:22:44+00:00] INFO: template[/opt/gitlab/sv/logrotate/log/config] sending create action to ruby_block[verify_chown_persisted_on_logrotate] (immediate)
3730 * ruby_block[verify_chown_persisted_on_logrotate] action create[2023-07-20T03:22:44+00:00] INFO: ruby_block[verify_chown_persisted_on_logrotate] called
3731 - execute the ruby block verify_chown_persisted_on_logrotate
3732 * ruby_block[verify_chown_persisted_on_logrotate] action nothing (skipped due to action :nothing)
3733 * link[/var/log/gitlab/logrotate/config] action create[2023-07-20T03:22:44+00:00] INFO: link[/var/log/gitlab/logrotate/config] created
3734 - create symlink at /var/log/gitlab/logrotate/config to /opt/gitlab/sv/logrotate/log/config
3735 * template[/opt/gitlab/sv/logrotate/log/run] action create[2023-07-20T03:22:44+00:00] INFO: template[/opt/gitlab/sv/logrotate/log/run] created file /opt/gitlab/sv/logrotate/log/run
3736 - create new file /opt/gitlab/sv/logrotate/log/run[2023-07-20T03:22:44+00:00] INFO: template[/opt/gitlab/sv/logrotate/log/run] updated file contents /opt/gitlab/sv/logrotate/log/run
3737 - update content in file /opt/gitlab/sv/logrotate/log/run from none to 637335
3738 --- /opt/gitlab/sv/logrotate/log/run 2023-07-20 03:22:44.000000000 +0000
3739 +++ /opt/gitlab/sv/logrotate/log/.chef-run20230720-37-ahf97n 2023-07-20 03:22:44.000000000 +0000
3740 @@ -1,5 +1,10 @@
3741 +#!/bin/sh
3742 +exec chpst -P \
3743 + -U root:root \
3744 + -u root:root \
3745 + svlogd -tt /var/log/gitlab/logrotate[2023-07-20T03:22:44+00:00] INFO: template[/opt/gitlab/sv/logrotate/log/run] owner changed to 0
3746[2023-07-20T03:22:44+00:00] INFO: template[/opt/gitlab/sv/logrotate/log/run] group changed to 0
3747[2023-07-20T03:22:44+00:00] INFO: template[/opt/gitlab/sv/logrotate/log/run] mode changed to 755
3748 - change mode from '' to '0755'
3749 - change owner from '' to 'root'
3750 - change group from '' to 'root'
3751 * directory[/opt/gitlab/sv/logrotate/env] action create[2023-07-20T03:22:44+00:00] INFO: directory[/opt/gitlab/sv/logrotate/env] created directory /opt/gitlab/sv/logrotate/env
3752 - create new directory /opt/gitlab/sv/logrotate/env[2023-07-20T03:22:44+00:00] INFO: directory[/opt/gitlab/sv/logrotate/env] owner changed to 0
3753[2023-07-20T03:22:44+00:00] INFO: directory[/opt/gitlab/sv/logrotate/env] group changed to 0
3754[2023-07-20T03:22:44+00:00] INFO: directory[/opt/gitlab/sv/logrotate/env] mode changed to 755
3755 - change mode from '' to '0755'
3756 - change owner from '' to 'root'
3757 - change group from '' to 'root'
3758 * ruby_block[Delete unmanaged env files for logrotate service] action run (skipped due to only_if)
3759 * template[/opt/gitlab/sv/logrotate/check] action create (skipped due to only_if)
3760 * template[/opt/gitlab/sv/logrotate/finish] action create (skipped due to only_if)
3761 * directory[/opt/gitlab/sv/logrotate/control] action create[2023-07-20T03:22:44+00:00] INFO: directory[/opt/gitlab/sv/logrotate/control] created directory /opt/gitlab/sv/logrotate/control
3762 - create new directory /opt/gitlab/sv/logrotate/control[2023-07-20T03:22:44+00:00] INFO: directory[/opt/gitlab/sv/logrotate/control] owner changed to 0
3763[2023-07-20T03:22:44+00:00] INFO: directory[/opt/gitlab/sv/logrotate/control] group changed to 0
3764[2023-07-20T03:22:44+00:00] INFO: directory[/opt/gitlab/sv/logrotate/control] mode changed to 755
3765 - change mode from '' to '0755'
3766 - change owner from '' to 'root'
3767 - change group from '' to 'root'
3768 * template[/opt/gitlab/sv/logrotate/control/t] action create[2023-07-20T03:22:44+00:00] INFO: template[/opt/gitlab/sv/logrotate/control/t] created file /opt/gitlab/sv/logrotate/control/t
3769 - create new file /opt/gitlab/sv/logrotate/control/t[2023-07-20T03:22:44+00:00] INFO: template[/opt/gitlab/sv/logrotate/control/t] updated file contents /opt/gitlab/sv/logrotate/control/t
3770 - update content in file /opt/gitlab/sv/logrotate/control/t from none to 8fa3fa
3771 --- /opt/gitlab/sv/logrotate/control/t 2023-07-20 03:22:44.000000000 +0000
3772 +++ /opt/gitlab/sv/logrotate/control/.chef-t20230720-37-550lk7 2023-07-20 03:22:44.000000000 +0000
3773 @@ -1,3 +1,6 @@
3774 +#!/bin/sh
3775 +echo "Received TERM from runit, sending to process group (-PID)"
3776 +kill -- -$(cat /opt/gitlab/service/logrotate/supervise/pid)[2023-07-20T03:22:44+00:00] INFO: template[/opt/gitlab/sv/logrotate/control/t] owner changed to 0
3777[2023-07-20T03:22:44+00:00] INFO: template[/opt/gitlab/sv/logrotate/control/t] group changed to 0
3778[2023-07-20T03:22:44+00:00] INFO: template[/opt/gitlab/sv/logrotate/control/t] mode changed to 755
3779 - change mode from '' to '0755'
3780 - change owner from '' to 'root'
3781 - change group from '' to 'root'
3782 * link[/opt/gitlab/init/logrotate] action create[2023-07-20T03:22:44+00:00] INFO: link[/opt/gitlab/init/logrotate] created
3783 - create symlink at /opt/gitlab/init/logrotate to /opt/gitlab/embedded/bin/sv
3784 * file[/opt/gitlab/sv/logrotate/down] action nothing (skipped due to action :nothing)
3785[2023-07-20T03:22:44+00:00] INFO: template[/opt/gitlab/sv/logrotate/run] sending run action to ruby_block[restart_service] (delayed)
3786 * ruby_block[restart_service] action run (skipped due to only_if)
3787[2023-07-20T03:22:44+00:00] INFO: template[/opt/gitlab/sv/logrotate/log/config] sending create action to ruby_block[reload_log_service] (delayed)
3788 * ruby_block[reload_log_service] action create
3789 * ruby_block[restart_service] action nothing (skipped due to action :nothing)
3790 * ruby_block[restart_log_service] action nothing (skipped due to action :nothing)
3791 * ruby_block[reload_log_service] action nothing (skipped due to action :nothing)
3792 * directory[/opt/gitlab/sv/logrotate] action create (up to date)
3793 * template[/opt/gitlab/sv/logrotate/run] action create (up to date)
3794 * directory[/opt/gitlab/sv/logrotate/log] action create (up to date)
3795 * directory[/opt/gitlab/sv/logrotate/log/main] action create (up to date)
3796 * template[/opt/gitlab/sv/logrotate/log/config] action create (up to date)
3797 * ruby_block[verify_chown_persisted_on_logrotate] action nothing (skipped due to action :nothing)
3798 * link[/var/log/gitlab/logrotate/config] action create (up to date)
3799 * template[/opt/gitlab/sv/logrotate/log/run] action create (up to date)
3800 * directory[/opt/gitlab/sv/logrotate/env] action create (up to date)
3801 * ruby_block[Delete unmanaged env files for logrotate service] action run (skipped due to only_if)
3802 * template[/opt/gitlab/sv/logrotate/check] action create (skipped due to only_if)
3803 * template[/opt/gitlab/sv/logrotate/finish] action create (skipped due to only_if)
3804 * directory[/opt/gitlab/sv/logrotate/control] action create (up to date)
3805 * template[/opt/gitlab/sv/logrotate/control/t] action create (up to date)
3806 * link[/opt/gitlab/init/logrotate] action create (up to date)
3807 * file[/opt/gitlab/sv/logrotate/down] action nothing (skipped due to action :nothing)
3808 * directory[/opt/gitlab/service] action create (up to date)
3809 * link[/opt/gitlab/service/logrotate] action create[2023-07-20T03:22:44+00:00] INFO: link[/opt/gitlab/service/logrotate] created
3810 - create symlink at /opt/gitlab/service/logrotate to /opt/gitlab/sv/logrotate
3811 * ruby_block[wait for logrotate service socket] action run[2023-07-20T03:22:46+00:00] INFO: ruby_block[wait for logrotate service socket] called
3812 - execute the ruby block wait for logrotate service socket
3813 * file[/var/log/gitlab/logrotate/current] action touch (skipped due to only_if)
3814[2023-07-20T03:22:46+00:00] INFO: ruby_block[reload_log_service] called
3815 - execute the ruby block reload_log_service
3816[2023-07-20T03:22:46+00:00] INFO: template[/opt/gitlab/sv/logrotate/log/run] sending run action to ruby_block[restart_log_service] (delayed)
3817 * ruby_block[restart_log_service] action run
3818 * ruby_block[restart_service] action nothing (skipped due to action :nothing)
3819 * ruby_block[restart_log_service] action nothing (skipped due to action :nothing)
3820 * ruby_block[reload_log_service] action nothing (skipped due to action :nothing)
3821 * directory[/opt/gitlab/sv/logrotate] action create (up to date)
3822 * template[/opt/gitlab/sv/logrotate/run] action create (up to date)
3823 * directory[/opt/gitlab/sv/logrotate/log] action create (up to date)
3824 * directory[/opt/gitlab/sv/logrotate/log/main] action create (up to date)
3825 * template[/opt/gitlab/sv/logrotate/log/config] action create (up to date)
3826 * ruby_block[verify_chown_persisted_on_logrotate] action nothing (skipped due to action :nothing)
3827 * link[/var/log/gitlab/logrotate/config] action create (up to date)
3828 * template[/opt/gitlab/sv/logrotate/log/run] action create (up to date)
3829 * directory[/opt/gitlab/sv/logrotate/env] action create (up to date)
3830 * ruby_block[Delete unmanaged env files for logrotate service] action run (skipped due to only_if)
3831 * template[/opt/gitlab/sv/logrotate/check] action create (skipped due to only_if)
3832 * template[/opt/gitlab/sv/logrotate/finish] action create (skipped due to only_if)
3833 * directory[/opt/gitlab/sv/logrotate/control] action create (up to date)
3834 * template[/opt/gitlab/sv/logrotate/control/t] action create (up to date)
3835 * link[/opt/gitlab/init/logrotate] action create (up to date)
3836 * file[/opt/gitlab/sv/logrotate/down] action nothing (skipped due to action :nothing)
3837 * directory[/opt/gitlab/service] action create (up to date)
3838 * link[/opt/gitlab/service/logrotate] action create (up to date)
3839 * ruby_block[wait for logrotate service socket] action run (skipped due to not_if)
3840 * file[/var/log/gitlab/logrotate/current] action touch (skipped due to only_if)
3841[2023-07-20T03:22:47+00:00] INFO: ruby_block[restart_log_service] called
3842 - execute the ruby block restart_log_service
3843 * directory[/opt/gitlab/service] action create (up to date)
3844 * link[/opt/gitlab/service/logrotate] action create (up to date)
3845 * ruby_block[wait for logrotate service socket] action run (skipped due to not_if)
3846 * file[/var/log/gitlab/logrotate/current] action touch (skipped due to only_if)
3847
3848 * execute[/opt/gitlab/bin/gitlab-ctl start logrotate] action run
3849 [execute] ok: run: logrotate: (pid 498) 2s
3850[2023-07-20T03:22:48+00:00] INFO: execute[/opt/gitlab/bin/gitlab-ctl start logrotate] ran successfully
3851 - execute /opt/gitlab/bin/gitlab-ctl start logrotate
3852Recipe: redis::disable
3853 * service[redis] action nothing (skipped due to action :nothing)
3854 * runit_service[redis] action disable
3855 * ruby_block[disable redis] action run (skipped due to only_if)
3856 (up to date)
3857Recipe: gitaly::disable
3858 * service[gitaly] action nothing (skipped due to action :nothing)
3859 * runit_service[gitaly] action disable
3860 * ruby_block[disable gitaly] action run (skipped due to only_if)
3861 (up to date)
3862 * consul_service[gitaly] action delete
3863 * file[/var/opt/gitlab/consul/config.d/gitaly-service.json] action delete (up to date)
3864 (up to date)
3865Recipe: postgresql::bin
3866 * ruby_block[check_postgresql_version] action run (skipped due to not_if)
3867 * ruby_block[check_postgresql_version_is_deprecated] action run (skipped due to not_if)
3868 * ruby_block[Link postgresql bin files to the correct version] action run[2023-07-20T03:22:48+00:00] INFO: ruby_block[Link postgresql bin files to the correct version] called
3869 - execute the ruby block Link postgresql bin files to the correct version
3870 * template[/opt/gitlab/etc/gitlab-psql-rc] action create[2023-07-20T03:22:48+00:00] INFO: template[/opt/gitlab/etc/gitlab-psql-rc] created file /opt/gitlab/etc/gitlab-psql-rc
3871 - create new file /opt/gitlab/etc/gitlab-psql-rc[2023-07-20T03:22:48+00:00] INFO: template[/opt/gitlab/etc/gitlab-psql-rc] updated file contents /opt/gitlab/etc/gitlab-psql-rc
3872 - update content in file /opt/gitlab/etc/gitlab-psql-rc from none to b7b8fc
3873 - suppressed sensitive resource[2023-07-20T03:22:48+00:00] INFO: template[/opt/gitlab/etc/gitlab-psql-rc] owner changed to 0
3874[2023-07-20T03:22:48+00:00] INFO: template[/opt/gitlab/etc/gitlab-psql-rc] group changed to 0
3875 - change owner from '' to 'root'
3876 - change group from '' to 'root'
3877Recipe: postgresql::disable
3878 * service[postgresql] action nothing (skipped due to action :nothing)
3879 * runit_service[postgresql] action disable
3880 * ruby_block[disable postgresql] action run (skipped due to only_if)
3881 (up to date)
3882Recipe: praefect::enable
3883 * directory[/var/opt/gitlab/praefect] action create[2023-07-20T03:22:48+00:00] INFO: directory[/var/opt/gitlab/praefect] created directory /var/opt/gitlab/praefect
3884 - create new directory /var/opt/gitlab/praefect[2023-07-20T03:22:48+00:00] INFO: directory[/var/opt/gitlab/praefect] owner changed to 998
3885[2023-07-20T03:22:48+00:00] INFO: directory[/var/opt/gitlab/praefect] mode changed to 700
3886 - change mode from '' to '0700'
3887 - change owner from '' to 'git'
3888 * directory[/var/log/gitlab/praefect] action create[2023-07-20T03:22:48+00:00] INFO: directory[/var/log/gitlab/praefect] created directory /var/log/gitlab/praefect
3889 - create new directory /var/log/gitlab/praefect[2023-07-20T03:22:48+00:00] INFO: directory[/var/log/gitlab/praefect] owner changed to 998
3890[2023-07-20T03:22:48+00:00] INFO: directory[/var/log/gitlab/praefect] mode changed to 700
3891 - change mode from '' to '0700'
3892 - change owner from '' to 'git'
3893 * env_dir[/opt/gitlab/etc/praefect/env] action create
3894 * directory[/opt/gitlab/etc/praefect/env] action create[2023-07-20T03:22:48+00:00] INFO: directory[/opt/gitlab/etc/praefect/env] created directory /opt/gitlab/etc/praefect/env
3895 - create new directory /opt/gitlab/etc/praefect/env
3896 * file[/opt/gitlab/etc/praefect/env/SSL_CERT_DIR] action create[2023-07-20T03:22:48+00:00] INFO: file[/opt/gitlab/etc/praefect/env/SSL_CERT_DIR] created file /opt/gitlab/etc/praefect/env/SSL_CERT_DIR
3897 - create new file /opt/gitlab/etc/praefect/env/SSL_CERT_DIR[2023-07-20T03:22:48+00:00] INFO: file[/opt/gitlab/etc/praefect/env/SSL_CERT_DIR] updated file contents /opt/gitlab/etc/praefect/env/SSL_CERT_DIR
3898 - update content in file /opt/gitlab/etc/praefect/env/SSL_CERT_DIR from none to 4f45cf
3899 - suppressed sensitive resource
3900 * file[/opt/gitlab/etc/praefect/env/GITALY_PID_FILE] action create[2023-07-20T03:22:48+00:00] INFO: file[/opt/gitlab/etc/praefect/env/GITALY_PID_FILE] created file /opt/gitlab/etc/praefect/env/GITALY_PID_FILE
3901 - create new file /opt/gitlab/etc/praefect/env/GITALY_PID_FILE[2023-07-20T03:22:48+00:00] INFO: file[/opt/gitlab/etc/praefect/env/GITALY_PID_FILE] updated file contents /opt/gitlab/etc/praefect/env/GITALY_PID_FILE
3902 - update content in file /opt/gitlab/etc/praefect/env/GITALY_PID_FILE from none to eb1c86
3903 - suppressed sensitive resource
3904 * file[/opt/gitlab/etc/praefect/env/WRAPPER_JSON_LOGGING] action create[2023-07-20T03:22:48+00:00] INFO: file[/opt/gitlab/etc/praefect/env/WRAPPER_JSON_LOGGING] created file /opt/gitlab/etc/praefect/env/WRAPPER_JSON_LOGGING
3905 - create new file /opt/gitlab/etc/praefect/env/WRAPPER_JSON_LOGGING[2023-07-20T03:22:48+00:00] INFO: file[/opt/gitlab/etc/praefect/env/WRAPPER_JSON_LOGGING] updated file contents /opt/gitlab/etc/praefect/env/WRAPPER_JSON_LOGGING
3906 - update content in file /opt/gitlab/etc/praefect/env/WRAPPER_JSON_LOGGING from none to b5bea4
3907 - suppressed sensitive resource
3908
3909 * template[Create praefect config.toml] action create[2023-07-20T03:22:48+00:00] INFO: template[Create praefect config.toml] created file /var/opt/gitlab/praefect/config.toml
3910 - create new file /var/opt/gitlab/praefect/config.toml[2023-07-20T03:22:48+00:00] INFO: template[Create praefect config.toml] updated file contents /var/opt/gitlab/praefect/config.toml
3911 - update content in file /var/opt/gitlab/praefect/config.toml from none to cc91d5
3912 - suppressed sensitive resource[2023-07-20T03:22:48+00:00] INFO: template[Create praefect config.toml] owner changed to 0
3913[2023-07-20T03:22:48+00:00] INFO: template[Create praefect config.toml] group changed to 998
3914[2023-07-20T03:22:48+00:00] INFO: template[Create praefect config.toml] mode changed to 640
3915 - change mode from '' to '0640'
3916 - change owner from '' to 'root'
3917 - change group from '' to 'git'
3918 * service[praefect] action nothing (skipped due to action :nothing)
3919 * runit_service[praefect] action enable
3920 * ruby_block[restart_service] action nothing (skipped due to action :nothing)
3921 * ruby_block[restart_log_service] action nothing (skipped due to action :nothing)
3922 * ruby_block[reload_log_service] action nothing (skipped due to action :nothing)
3923 * directory[/opt/gitlab/sv/praefect] action create[2023-07-20T03:22:48+00:00] INFO: directory[/opt/gitlab/sv/praefect] created directory /opt/gitlab/sv/praefect
3924 - create new directory /opt/gitlab/sv/praefect[2023-07-20T03:22:48+00:00] INFO: directory[/opt/gitlab/sv/praefect] owner changed to 0
3925[2023-07-20T03:22:48+00:00] INFO: directory[/opt/gitlab/sv/praefect] group changed to 0
3926[2023-07-20T03:22:48+00:00] INFO: directory[/opt/gitlab/sv/praefect] mode changed to 755
3927 - change mode from '' to '0755'
3928 - change owner from '' to 'root'
3929 - change group from '' to 'root'
3930 * template[/opt/gitlab/sv/praefect/run] action create[2023-07-20T03:22:48+00:00] INFO: template[/opt/gitlab/sv/praefect/run] created file /opt/gitlab/sv/praefect/run
3931 - create new file /opt/gitlab/sv/praefect/run[2023-07-20T03:22:48+00:00] INFO: template[/opt/gitlab/sv/praefect/run] updated file contents /opt/gitlab/sv/praefect/run
3932 - update content in file /opt/gitlab/sv/praefect/run from none to 3c4831
3933 --- /opt/gitlab/sv/praefect/run 2023-07-20 03:22:48.000000000 +0000
3934 +++ /opt/gitlab/sv/praefect/.chef-run20230720-37-rwr98f 2023-07-20 03:22:48.000000000 +0000
3935 @@ -1,12 +1,24 @@
3936 +#!/bin/sh
3937 +set -e # fail on errors
3938 +
3939 +# Redirect stderr -> stdout
3940 +exec 2>&1
3941 +
3942 +cd /var/opt/gitlab/praefect
3943 +
3944 +exec chpst -e /opt/gitlab/etc/praefect/env -P \
3945 + -U git:git \
3946 + -u git:git \
3947 + /opt/gitlab/embedded/bin/gitaly-wrapper /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml[2023-07-20T03:22:48+00:00] INFO: template[/opt/gitlab/sv/praefect/run] owner changed to 0
3948[2023-07-20T03:22:48+00:00] INFO: template[/opt/gitlab/sv/praefect/run] group changed to 0
3949[2023-07-20T03:22:48+00:00] INFO: template[/opt/gitlab/sv/praefect/run] mode changed to 755
3950 - change mode from '' to '0755'
3951 - change owner from '' to 'root'
3952 - change group from '' to 'root'
3953 * directory[/opt/gitlab/sv/praefect/log] action create[2023-07-20T03:22:48+00:00] INFO: directory[/opt/gitlab/sv/praefect/log] created directory /opt/gitlab/sv/praefect/log
3954 - create new directory /opt/gitlab/sv/praefect/log[2023-07-20T03:22:48+00:00] INFO: directory[/opt/gitlab/sv/praefect/log] owner changed to 0
3955[2023-07-20T03:22:48+00:00] INFO: directory[/opt/gitlab/sv/praefect/log] group changed to 0
3956[2023-07-20T03:22:48+00:00] INFO: directory[/opt/gitlab/sv/praefect/log] mode changed to 755
3957 - change mode from '' to '0755'
3958 - change owner from '' to 'root'
3959 - change group from '' to 'root'
3960 * directory[/opt/gitlab/sv/praefect/log/main] action create[2023-07-20T03:22:48+00:00] INFO: directory[/opt/gitlab/sv/praefect/log/main] created directory /opt/gitlab/sv/praefect/log/main
3961 - create new directory /opt/gitlab/sv/praefect/log/main[2023-07-20T03:22:48+00:00] INFO: directory[/opt/gitlab/sv/praefect/log/main] owner changed to 0
3962[2023-07-20T03:22:48+00:00] INFO: directory[/opt/gitlab/sv/praefect/log/main] group changed to 0
3963[2023-07-20T03:22:48+00:00] INFO: directory[/opt/gitlab/sv/praefect/log/main] mode changed to 755
3964 - change mode from '' to '0755'
3965 - change owner from '' to 'root'
3966 - change group from '' to 'root'
3967 * template[/opt/gitlab/sv/praefect/log/config] action create[2023-07-20T03:22:48+00:00] INFO: template[/opt/gitlab/sv/praefect/log/config] created file /opt/gitlab/sv/praefect/log/config
3968 - create new file /opt/gitlab/sv/praefect/log/config[2023-07-20T03:22:48+00:00] INFO: template[/opt/gitlab/sv/praefect/log/config] updated file contents /opt/gitlab/sv/praefect/log/config
3969 - update content in file /opt/gitlab/sv/praefect/log/config from none to 623c00
3970 --- /opt/gitlab/sv/praefect/log/config 2023-07-20 03:22:48.000000000 +0000
3971 +++ /opt/gitlab/sv/praefect/log/.chef-config20230720-37-tdbp3 2023-07-20 03:22:48.000000000 +0000
3972 @@ -1,6 +1,12 @@
3973 +s209715200
3974 +n30
3975 +t86400
3976 +!gzip
3977 +
3978 +[2023-07-20T03:22:48+00:00] INFO: template[/opt/gitlab/sv/praefect/log/config] owner changed to 0
3979[2023-07-20T03:22:48+00:00] INFO: template[/opt/gitlab/sv/praefect/log/config] group changed to 0
3980[2023-07-20T03:22:48+00:00] INFO: template[/opt/gitlab/sv/praefect/log/config] mode changed to 644
3981 - change mode from '' to '0644'
3982 - change owner from '' to 'root'
3983 - change group from '' to 'root'
3984[2023-07-20T03:22:48+00:00] INFO: template[/opt/gitlab/sv/praefect/log/config] sending create action to ruby_block[verify_chown_persisted_on_praefect] (immediate)
3985 * ruby_block[verify_chown_persisted_on_praefect] action create[2023-07-20T03:22:48+00:00] INFO: ruby_block[verify_chown_persisted_on_praefect] called
3986 - execute the ruby block verify_chown_persisted_on_praefect
3987 * ruby_block[verify_chown_persisted_on_praefect] action nothing (skipped due to action :nothing)
3988 * link[/var/log/gitlab/praefect/config] action create[2023-07-20T03:22:48+00:00] INFO: link[/var/log/gitlab/praefect/config] created
3989 - create symlink at /var/log/gitlab/praefect/config to /opt/gitlab/sv/praefect/log/config
3990 * template[/opt/gitlab/sv/praefect/log/run] action create[2023-07-20T03:22:48+00:00] INFO: template[/opt/gitlab/sv/praefect/log/run] created file /opt/gitlab/sv/praefect/log/run
3991 - create new file /opt/gitlab/sv/praefect/log/run[2023-07-20T03:22:48+00:00] INFO: template[/opt/gitlab/sv/praefect/log/run] updated file contents /opt/gitlab/sv/praefect/log/run
3992 - update content in file /opt/gitlab/sv/praefect/log/run from none to 94f1c2
3993 --- /opt/gitlab/sv/praefect/log/run 2023-07-20 03:22:48.000000000 +0000
3994 +++ /opt/gitlab/sv/praefect/log/.chef-run20230720-37-ur38li 2023-07-20 03:22:48.000000000 +0000
3995 @@ -1,5 +1,10 @@
3996 +#!/bin/sh
3997 +exec chpst -P \
3998 + -U root:root \
3999 + -u root:root \
4000 + svlogd /var/log/gitlab/praefect[2023-07-20T03:22:48+00:00] INFO: template[/opt/gitlab/sv/praefect/log/run] owner changed to 0
4001[2023-07-20T03:22:48+00:00] INFO: template[/opt/gitlab/sv/praefect/log/run] group changed to 0
4002[2023-07-20T03:22:48+00:00] INFO: template[/opt/gitlab/sv/praefect/log/run] mode changed to 755
4003 - change mode from '' to '0755'
4004 - change owner from '' to 'root'
4005 - change group from '' to 'root'
4006 * directory[/opt/gitlab/sv/praefect/env] action create[2023-07-20T03:22:48+00:00] INFO: directory[/opt/gitlab/sv/praefect/env] created directory /opt/gitlab/sv/praefect/env
4007 - create new directory /opt/gitlab/sv/praefect/env[2023-07-20T03:22:48+00:00] INFO: directory[/opt/gitlab/sv/praefect/env] owner changed to 0
4008[2023-07-20T03:22:48+00:00] INFO: directory[/opt/gitlab/sv/praefect/env] group changed to 0
4009[2023-07-20T03:22:48+00:00] INFO: directory[/opt/gitlab/sv/praefect/env] mode changed to 755
4010 - change mode from '' to '0755'
4011 - change owner from '' to 'root'
4012 - change group from '' to 'root'
4013 * ruby_block[Delete unmanaged env files for praefect service] action run (skipped due to only_if)
4014 * template[/opt/gitlab/sv/praefect/check] action create (skipped due to only_if)
4015 * template[/opt/gitlab/sv/praefect/finish] action create (skipped due to only_if)
4016 * directory[/opt/gitlab/sv/praefect/control] action create[2023-07-20T03:22:48+00:00] INFO: directory[/opt/gitlab/sv/praefect/control] created directory /opt/gitlab/sv/praefect/control
4017 - create new directory /opt/gitlab/sv/praefect/control[2023-07-20T03:22:48+00:00] INFO: directory[/opt/gitlab/sv/praefect/control] owner changed to 0
4018[2023-07-20T03:22:48+00:00] INFO: directory[/opt/gitlab/sv/praefect/control] group changed to 0
4019[2023-07-20T03:22:48+00:00] INFO: directory[/opt/gitlab/sv/praefect/control] mode changed to 755
4020 - change mode from '' to '0755'
4021 - change owner from '' to 'root'
4022 - change group from '' to 'root'
4023 * link[/opt/gitlab/init/praefect] action create[2023-07-20T03:22:48+00:00] INFO: link[/opt/gitlab/init/praefect] created
4024 - create symlink at /opt/gitlab/init/praefect to /opt/gitlab/embedded/bin/sv
4025 * file[/opt/gitlab/sv/praefect/down] action nothing (skipped due to action :nothing)
4026[2023-07-20T03:22:48+00:00] INFO: template[/opt/gitlab/sv/praefect/run] sending run action to ruby_block[restart_service] (delayed)
4027 * ruby_block[restart_service] action run (skipped due to only_if)
4028[2023-07-20T03:22:48+00:00] INFO: template[/opt/gitlab/sv/praefect/log/config] sending create action to ruby_block[reload_log_service] (delayed)
4029 * ruby_block[reload_log_service] action create
4030 * ruby_block[restart_service] action nothing (skipped due to action :nothing)
4031 * ruby_block[restart_log_service] action nothing (skipped due to action :nothing)
4032 * ruby_block[reload_log_service] action nothing (skipped due to action :nothing)
4033 * directory[/opt/gitlab/sv/praefect] action create (up to date)
4034 * template[/opt/gitlab/sv/praefect/run] action create (up to date)
4035 * directory[/opt/gitlab/sv/praefect/log] action create (up to date)
4036 * directory[/opt/gitlab/sv/praefect/log/main] action create (up to date)
4037 * template[/opt/gitlab/sv/praefect/log/config] action create (up to date)
4038 * ruby_block[verify_chown_persisted_on_praefect] action nothing (skipped due to action :nothing)
4039 * link[/var/log/gitlab/praefect/config] action create (up to date)
4040 * template[/opt/gitlab/sv/praefect/log/run] action create (up to date)
4041 * directory[/opt/gitlab/sv/praefect/env] action create (up to date)
4042 * ruby_block[Delete unmanaged env files for praefect service] action run (skipped due to only_if)
4043 * template[/opt/gitlab/sv/praefect/check] action create (skipped due to only_if)
4044 * template[/opt/gitlab/sv/praefect/finish] action create (skipped due to only_if)
4045 * directory[/opt/gitlab/sv/praefect/control] action create (up to date)
4046 * link[/opt/gitlab/init/praefect] action create (up to date)
4047 * file[/opt/gitlab/sv/praefect/down] action nothing (skipped due to action :nothing)
4048 * directory[/opt/gitlab/service] action create (up to date)
4049 * link[/opt/gitlab/service/praefect] action create[2023-07-20T03:22:48+00:00] INFO: link[/opt/gitlab/service/praefect] created
4050 - create symlink at /opt/gitlab/service/praefect to /opt/gitlab/sv/praefect
4051 * ruby_block[wait for praefect service socket] action run[2023-07-20T03:22:52+00:00] INFO: ruby_block[wait for praefect service socket] called
4052 - execute the ruby block wait for praefect service socket
4053 * file[/var/log/gitlab/praefect/current] action touch (skipped due to only_if)
4054[2023-07-20T03:22:54+00:00] INFO: ruby_block[reload_log_service] called
4055 - execute the ruby block reload_log_service
4056[2023-07-20T03:22:54+00:00] INFO: template[/opt/gitlab/sv/praefect/log/run] sending run action to ruby_block[restart_log_service] (delayed)
4057 * ruby_block[restart_log_service] action run
4058 * ruby_block[restart_service] action nothing (skipped due to action :nothing)
4059 * ruby_block[restart_log_service] action nothing (skipped due to action :nothing)
4060 * ruby_block[reload_log_service] action nothing (skipped due to action :nothing)
4061 * directory[/opt/gitlab/sv/praefect] action create (up to date)
4062 * template[/opt/gitlab/sv/praefect/run] action create (up to date)
4063 * directory[/opt/gitlab/sv/praefect/log] action create (up to date)
4064 * directory[/opt/gitlab/sv/praefect/log/main] action create (up to date)
4065 * template[/opt/gitlab/sv/praefect/log/config] action create (up to date)
4066 * ruby_block[verify_chown_persisted_on_praefect] action nothing (skipped due to action :nothing)
4067 * link[/var/log/gitlab/praefect/config] action create (up to date)
4068 * template[/opt/gitlab/sv/praefect/log/run] action create (up to date)
4069 * directory[/opt/gitlab/sv/praefect/env] action create (up to date)
4070 * ruby_block[Delete unmanaged env files for praefect service] action run (skipped due to only_if)
4071 * template[/opt/gitlab/sv/praefect/check] action create (skipped due to only_if)
4072 * template[/opt/gitlab/sv/praefect/finish] action create (skipped due to only_if)
4073 * directory[/opt/gitlab/sv/praefect/control] action create (up to date)
4074 * link[/opt/gitlab/init/praefect] action create (up to date)
4075 * file[/opt/gitlab/sv/praefect/down] action nothing (skipped due to action :nothing)
4076 * directory[/opt/gitlab/service] action create (up to date)
4077 * link[/opt/gitlab/service/praefect] action create (up to date)
4078 * ruby_block[wait for praefect service socket] action run (skipped due to not_if)
4079 * file[/var/log/gitlab/praefect/current] action touch (skipped due to only_if)
4080[2023-07-20T03:22:54+00:00] INFO: ruby_block[restart_log_service] called
4081 - execute the ruby block restart_log_service
4082 * directory[/opt/gitlab/service] action create (up to date)
4083 * link[/opt/gitlab/service/praefect] action create (up to date)
4084 * ruby_block[wait for praefect service socket] action run (skipped due to not_if)
4085 * file[/var/log/gitlab/praefect/current] action touch (skipped due to only_if)
4086
4087 * execute[/opt/gitlab/bin/gitlab-ctl start praefect] action run
4088 [execute] ok: run: praefect: (pid 515) 3s
4089[2023-07-20T03:22:55+00:00] INFO: execute[/opt/gitlab/bin/gitlab-ctl start praefect] ran successfully
4090 - execute /opt/gitlab/bin/gitlab-ctl start praefect
4091 * version_file[Create Praefect version file] action create
4092 * file[/var/opt/gitlab/praefect/VERSION] action create[2023-07-20T03:22:55+00:00] INFO: file[/var/opt/gitlab/praefect/VERSION] created file /var/opt/gitlab/praefect/VERSION
4093 - create new file /var/opt/gitlab/praefect/VERSION[2023-07-20T03:22:55+00:00] INFO: file[/var/opt/gitlab/praefect/VERSION] updated file contents /var/opt/gitlab/praefect/VERSION
4094 - update content in file /var/opt/gitlab/praefect/VERSION from none to 9d85fe
4095 --- /var/opt/gitlab/praefect/VERSION 2023-07-20 03:22:55.000000000 +0000
4096 +++ /var/opt/gitlab/praefect/.chef-VERSION20230720-37-773mc2 2023-07-20 03:22:55.000000000 +0000
4097 @@ -1 +1,2 @@
4098 +Praefect, version 16.2.0-rc4-77-gaf0a9ba61
4099
4100[2023-07-20T03:22:55+00:00] INFO: version_file[Create Praefect version file] not queuing delayed action hup on runit_service[praefect] (delayed), as it's already been queued
4101 * consul_service[praefect] action delete
4102 * file[/var/opt/gitlab/consul/config.d/praefect-service.json] action delete (up to date)
4103 (up to date)
4104Recipe: praefect::database_migrations
4105 * bash[migrate praefect database] action run
4106 [execute] praefect sql-migrate: migrations to apply: 42
4107
4108 = 2023-07-20T03:22:55 20200109161404_hello_world: migrating
4109 == 2023-07-20T03:22:55 20200109161404_hello_world: applied (2.289672ms)
4110 = 2023-07-20T03:22:55 20200113151438_1_test_migration: migrating
4111 == 2023-07-20T03:22:55 20200113151438_1_test_migration: applied (1.833381ms)
4112 = 2023-07-20T03:22:55 20200224220728_job_queue: migrating
4113 == 2023-07-20T03:22:55 20200224220728_job_queue: applied (18.672773ms)
4114 = 2023-07-20T03:22:55 20200225220728_readd_job_queue: migrating
4115 == 2023-07-20T03:22:55 20200225220728_readd_job_queue: applied (3.366342ms)
4116 = 2023-07-20T03:22:55 20200324001604_add_sql_election_tables: migrating
4117 == 2023-07-20T03:22:55 20200324001604_add_sql_election_tables: applied (18.468332ms)
4118 = 2023-07-20T03:22:55 20200422131451_add_shard_read_only_column: migrating
4119 == 2023-07-20T03:22:55 20200422131451_add_shard_read_only_column: applied (1.779881ms)
4120 = 2023-07-20T03:22:55 20200512131219_replication_job_indexing: migrating
4121 == 2023-07-20T03:22:55 20200512131219_replication_job_indexing: applied (4.204202ms)
4122 = 2023-07-20T03:22:55 20200527103816_drop_old_gitaly_tables: migrating
4123 == 2023-07-20T03:22:55 20200527103816_drop_old_gitaly_tables: applied (2.039441ms)
4124 = 2023-07-20T03:22:55 20200602154246_remember_previous_writable_primary: migrating
4125 == 2023-07-20T03:22:55 20200602154246_remember_previous_writable_primary: applied (1.860471ms)
4126 = 2023-07-20T03:22:55 20200707101830_repositories_table: migrating
4127 == 2023-07-20T03:22:55 20200707101830_repositories_table: applied (11.237818ms)
4128 = 2023-07-20T03:22:55 20200810055650_replication_queue_cleanup: migrating
4129 == 2023-07-20T03:22:55 20200810055650_replication_queue_cleanup: applied (2.392063ms)
4130 = 2023-07-20T03:22:55 20200921154417_repositories_nullable_generation: migrating
4131 == 2023-07-20T03:22:55 20200921154417_repositories_nullable_generation: applied (1.882931ms)
4132 = 2023-07-20T03:22:55 20200921170311_repositories_primary_column: migrating
4133 == 2023-07-20T03:22:55 20200921170311_repositories_primary_column: applied (2.037801ms)
4134 = 2023-07-20T03:22:55 20201006125956_trigger_repository_update_generation: migrating
4135 == 2023-07-20T03:22:55 20201006125956_trigger_repository_update_generation: applied (3.519982ms)
4136 = 2023-07-20T03:22:55 20201102115118_variable_replication_factor: migrating
4137 == 2023-07-20T03:22:55 20201102115118_variable_replication_factor: applied (1.956651ms)
4138 = 2023-07-20T03:22:55 20201102171914_virtual_storage_configuration: migrating
4139 == 2023-07-20T03:22:55 20201102171914_virtual_storage_configuration: applied (7.042335ms)
4140 = 2023-07-20T03:22:55 20201126165633_repository_assignments_table: migrating
4141 == 2023-07-20T03:22:55 20201126165633_repository_assignments_table: applied (7.077985ms)
4142 = 2023-07-20T03:22:55 20201208163237_cleanup_notifiactions_payload: migrating
4143 == 2023-07-20T03:22:55 20201208163237_cleanup_notifiactions_payload: applied (2.143532ms)
4144 = 2023-07-20T03:22:55 20201231075619_remove_unused_assigned_column: migrating
4145 == 2023-07-20T03:22:55 20201231075619_remove_unused_assigned_column: applied (2.33373ms)
4146 = 2023-07-20T03:22:55 20210223130233_delete_replica_unique_index: migrating
4147 == 2023-07-20T03:22:55 20210223130233_delete_replica_unique_index: applied (6.631115ms)
4148 = 2023-07-20T03:22:55 20210225101159_replication_queue_target_index: migrating
4149 == 2023-07-20T03:22:55 20210225101159_replication_queue_target_index: applied (6.080865ms)
4150 = 2023-07-20T03:22:55 20210525143540_healthy_storages_view: migrating
4151 == 2023-07-20T03:22:55 20210525143540_healthy_storages_view: applied (4.072012ms)
4152 = 2023-07-20T03:22:55 20210525173505_valid_primaries_view: migrating
4153 == 2023-07-20T03:22:55 20210525173505_valid_primaries_view: applied (4.648384ms)
4154 = 2023-07-20T03:22:55 20210607124235_optimize_valid_primaries_view: migrating
4155 == 2023-07-20T03:22:55 20210607124235_optimize_valid_primaries_view: applied (3.620433ms)
4156 = 2023-07-20T03:22:55 20210727085659_repository_ids: migrating
4157 == 2023-07-20T03:22:55 20210727085659_repository_ids: applied (16.210291ms)
4158 = 2023-07-20T03:22:55 20210906130405_add_replica_path: migrating
4159 == 2023-07-20T03:22:55 20210906130405_add_replica_path: applied (2.328752ms)
4160 = 2023-07-20T03:22:55 20210906145021_link_repository_id: migrating
4161 == 2023-07-20T03:22:55 20210906145021_link_repository_id: applied (3.072412ms)
4162 = 2023-07-20T03:22:55 20210914115710_storage_cleanups_table: migrating
4163 == 2023-07-20T03:22:55 20210914115710_storage_cleanups_table: applied (7.204225ms)
4164 = 2023-07-20T03:22:55 20210921131816_backfill_replica_path: migrating
4165 == 2023-07-20T03:22:55 20210921131816_backfill_replica_path: applied (2.079641ms)
4166 = 2023-07-20T03:22:55 20210922091614_repository_id_primary_key_indexes: migrating
4167 == 2023-07-20T03:22:55 20210922091614_repository_id_primary_key_indexes: applied (6.696964ms)
4168 = 2023-07-20T03:22:55 20210922143752_storage_repositories_cascade_delete: migrating
4169 == 2023-07-20T03:22:55 20210922143752_storage_repositories_cascade_delete: applied (2.989753ms)
4170 = 2023-07-20T03:22:55 20210923155827_valid_primaries_view_repository_id: migrating
4171 == 2023-07-20T03:22:55 20210923155827_valid_primaries_view_repository_id: applied (3.795363ms)
4172 = 2023-07-20T03:22:55 20210927083631_repository_path_index: migrating
4173 == 2023-07-20T03:22:55 20210927083631_repository_path_index: applied (6.250546ms)
4174 = 2023-07-20T03:22:55 20211018142158_valid_primaries_view_join_repo_id: migrating
4175 == 2023-07-20T03:22:55 20211018142158_valid_primaries_view_join_repo_id: applied (3.691261ms)
4176 = 2023-07-20T03:22:55 20211025113445_remove_cancelled_replication_events: migrating
4177 == 2023-07-20T03:22:55 20211025113445_remove_cancelled_replication_events: applied (2.490943ms)
4178 = 2023-07-20T03:22:55 20211105100456_check_non_null_repository_id: migrating
4179 == 2023-07-20T03:22:55 20211105100456_check_non_null_repository_id: applied (2.563841ms)
4180 = 2023-07-20T03:22:55 20211105100458_non_nullable_repository_id: migrating
4181 == 2023-07-20T03:22:55 20211105100458_non_nullable_repository_id: applied (3.228264ms)
4182 = 2023-07-20T03:22:55 20220215160117_replication_queue_cleanup_again: migrating
4183 == 2023-07-20T03:22:55 20220215160117_replication_queue_cleanup_again: applied (2.249781ms)
4184 = 2023-07-20T03:22:55 20220303105110_background_verification_columns: migrating
4185 == 2023-07-20T03:22:55 20220303105110_background_verification_columns: applied (5.304193ms)
4186 = 2023-07-20T03:22:55 20220404131105_stale_verification_lease_index: migrating
4187 == 2023-07-20T03:22:55 20220404131105_stale_verification_lease_index: applied (4.893943ms)
4188 = 2023-07-20T03:22:55 20220407111624_ignore_verification_in_triggers: migrating
4189 == 2023-07-20T03:22:55 20220407111624_ignore_verification_in_triggers: applied (2.741962ms)
4190 = 2023-07-20T03:22:55 20220520083313_remove_maintenance_replication_events: migrating
4191 == 2023-07-20T03:22:55 20220520083313_remove_maintenance_replication_events: applied (3.642621ms)
4192
4193 praefect sql-migrate: OK (applied 42 migrations)
4194[2023-07-20T03:22:55+00:00] INFO: bash[migrate praefect database] ran successfully
4195 - execute "bash"
4196[2023-07-20T03:22:55+00:00] INFO: bash[migrate praefect database] sending hup action to runit_service[praefect] (immediate)
4197Recipe: praefect::enable
4198 * runit_service[praefect] action hup[2023-07-20T03:22:55+00:00] INFO: runit_service[praefect] signalled (HUP)
4199[2023-07-20T03:22:55+00:00] INFO: runit_service[praefect] sent hup
4200 - send hup to runit_service[praefect]
4201Recipe: gitlab-kas::enable
4202 * directory[/var/opt/gitlab/gitlab-kas] action create[2023-07-20T03:22:55+00:00] INFO: directory[/var/opt/gitlab/gitlab-kas] created directory /var/opt/gitlab/gitlab-kas
4203 - create new directory /var/opt/gitlab/gitlab-kas[2023-07-20T03:22:55+00:00] INFO: directory[/var/opt/gitlab/gitlab-kas] owner changed to 998
4204[2023-07-20T03:22:55+00:00] INFO: directory[/var/opt/gitlab/gitlab-kas] mode changed to 700
4205 - change mode from '' to '0700'
4206 - change owner from '' to 'git'
4207 * directory[/opt/gitlab/etc/gitlab-kas] action create[2023-07-20T03:22:55+00:00] INFO: directory[/opt/gitlab/etc/gitlab-kas] created directory /opt/gitlab/etc/gitlab-kas
4208 - create new directory /opt/gitlab/etc/gitlab-kas[2023-07-20T03:22:55+00:00] INFO: directory[/opt/gitlab/etc/gitlab-kas] owner changed to 998
4209[2023-07-20T03:22:55+00:00] INFO: directory[/opt/gitlab/etc/gitlab-kas] mode changed to 700
4210 - change mode from '' to '0700'
4211 - change owner from '' to 'git'
4212 * directory[/var/log/gitlab/gitlab-kas] action create[2023-07-20T03:22:55+00:00] INFO: directory[/var/log/gitlab/gitlab-kas] created directory /var/log/gitlab/gitlab-kas
4213 - create new directory /var/log/gitlab/gitlab-kas[2023-07-20T03:22:55+00:00] INFO: directory[/var/log/gitlab/gitlab-kas] owner changed to 998
4214[2023-07-20T03:22:55+00:00] INFO: directory[/var/log/gitlab/gitlab-kas] mode changed to 700
4215 - change mode from '' to '0700'
4216 - change owner from '' to 'git'
4217 * ruby_block[websocket TLS termination] action run (skipped due to only_if)
4218 * version_file[Create version file for Gitlab KAS] action create
4219 * file[/var/opt/gitlab/gitlab-kas/VERSION] action create[2023-07-20T03:22:55+00:00] INFO: file[/var/opt/gitlab/gitlab-kas/VERSION] created file /var/opt/gitlab/gitlab-kas/VERSION
4220 - create new file /var/opt/gitlab/gitlab-kas/VERSION[2023-07-20T03:22:55+00:00] INFO: file[/var/opt/gitlab/gitlab-kas/VERSION] updated file contents /var/opt/gitlab/gitlab-kas/VERSION
4221 - update content in file /var/opt/gitlab/gitlab-kas/VERSION from none to d3f397
4222 --- /var/opt/gitlab/gitlab-kas/VERSION 2023-07-20 03:22:55.000000000 +0000
4223 +++ /var/opt/gitlab/gitlab-kas/.chef-VERSION20230720-37-t5rt9m 2023-07-20 03:22:55.000000000 +0000
4224 @@ -1 +1,2 @@
4225 +kas version v16.2.0, commit: 79f3b36c, built: 20230720.023210
4226
4227 * file[/var/opt/gitlab/gitlab-kas/authentication_secret_file] action create[2023-07-20T03:22:55+00:00] INFO: file[/var/opt/gitlab/gitlab-kas/authentication_secret_file] created file /var/opt/gitlab/gitlab-kas/authentication_secret_file
4228 - create new file /var/opt/gitlab/gitlab-kas/authentication_secret_file[2023-07-20T03:22:55+00:00] INFO: file[/var/opt/gitlab/gitlab-kas/authentication_secret_file] updated file contents /var/opt/gitlab/gitlab-kas/authentication_secret_file
4229 - update content in file /var/opt/gitlab/gitlab-kas/authentication_secret_file from none to 00e13d
4230 --- /var/opt/gitlab/gitlab-kas/authentication_secret_file 2023-07-20 03:22:55.000000000 +0000
4231 +++ /var/opt/gitlab/gitlab-kas/.chef-authentication_secret_file20230720-37-esdlfe 2023-07-20 03:22:55.000000000 +0000
4232 @@ -1 +1,2 @@
4233 +NDJkNWVkYzMwMjEwNzNlYzdjZmU4MWNmZGVmYmRiZTQ=[2023-07-20T03:22:55+00:00] INFO: file[/var/opt/gitlab/gitlab-kas/authentication_secret_file] owner changed to 0
4234[2023-07-20T03:22:55+00:00] INFO: file[/var/opt/gitlab/gitlab-kas/authentication_secret_file] group changed to 998
4235[2023-07-20T03:22:55+00:00] INFO: file[/var/opt/gitlab/gitlab-kas/authentication_secret_file] mode changed to 640
4236 - change mode from '' to '0640'
4237 - change owner from '' to 'root'
4238 - change group from '' to 'git'
4239[2023-07-20T03:22:55+00:00] INFO: file[/var/opt/gitlab/gitlab-kas/authentication_secret_file] not queuing delayed action restart on runit_service[gitlab-kas] (delayed), as it's already been queued
4240 * file[/var/opt/gitlab/gitlab-kas/private_api_authentication_secret_file] action create[2023-07-20T03:22:56+00:00] INFO: file[/var/opt/gitlab/gitlab-kas/private_api_authentication_secret_file] created file /var/opt/gitlab/gitlab-kas/private_api_authentication_secret_file
4241 - create new file /var/opt/gitlab/gitlab-kas/private_api_authentication_secret_file[2023-07-20T03:22:56+00:00] INFO: file[/var/opt/gitlab/gitlab-kas/private_api_authentication_secret_file] updated file contents /var/opt/gitlab/gitlab-kas/private_api_authentication_secret_file
4242 - update content in file /var/opt/gitlab/gitlab-kas/private_api_authentication_secret_file from none to 1a1717
4243 --- /var/opt/gitlab/gitlab-kas/private_api_authentication_secret_file 2023-07-20 03:22:56.000000000 +0000
4244 +++ /var/opt/gitlab/gitlab-kas/.chef-private_api_authentication_secret_file20230720-37-kvwnhq 2023-07-20 03:22:56.000000000 +0000
4245 @@ -1 +1,2 @@
4246 +YTIwOWI2NTc5ZGRjOTcxODZiN2U3NGMwZjdjY2ZhNjE=[2023-07-20T03:22:56+00:00] INFO: file[/var/opt/gitlab/gitlab-kas/private_api_authentication_secret_file] owner changed to 0
4247[2023-07-20T03:22:56+00:00] INFO: file[/var/opt/gitlab/gitlab-kas/private_api_authentication_secret_file] group changed to 998
4248[2023-07-20T03:22:56+00:00] INFO: file[/var/opt/gitlab/gitlab-kas/private_api_authentication_secret_file] mode changed to 640
4249 - change mode from '' to '0640'
4250 - change owner from '' to 'root'
4251 - change group from '' to 'git'
4252[2023-07-20T03:22:56+00:00] INFO: file[/var/opt/gitlab/gitlab-kas/private_api_authentication_secret_file] not queuing delayed action restart on runit_service[gitlab-kas] (delayed), as it's already been queued
4253 * file[/var/opt/gitlab/gitlab-kas/redis_password_file] action create (skipped due to only_if)
4254 * file[/var/opt/gitlab/gitlab-kas/redis_sentinels_password_file] action create (skipped due to only_if)
4255 * template[/var/opt/gitlab/gitlab-kas/gitlab-kas-config.yml] action create[2023-07-20T03:22:56+00:00] INFO: template[/var/opt/gitlab/gitlab-kas/gitlab-kas-config.yml] created file /var/opt/gitlab/gitlab-kas/gitlab-kas-config.yml
4256 - create new file /var/opt/gitlab/gitlab-kas/gitlab-kas-config.yml[2023-07-20T03:22:56+00:00] INFO: template[/var/opt/gitlab/gitlab-kas/gitlab-kas-config.yml] updated file contents /var/opt/gitlab/gitlab-kas/gitlab-kas-config.yml
4257 - update content in file /var/opt/gitlab/gitlab-kas/gitlab-kas-config.yml from none to 66c892
4258 --- /var/opt/gitlab/gitlab-kas/gitlab-kas-config.yml 2023-07-20 03:22:56.000000000 +0000
4259 +++ /var/opt/gitlab/gitlab-kas/.chef-gitlab-kas-config20230720-37-z0qq27.yml 2023-07-20 03:22:56.000000000 +0000
4260 @@ -1,44 +1,88 @@
4261 +agent:
4262 + listen:
4263 + address: localhost:8150
4264 + network: tcp
4265 + websocket: true
4266 + configuration:
4267 + poll_period: 20s
4268 + gitops:
4269 + poll_period: 20s
4270 + project_info_cache_ttl: 300s
4271 + project_info_cache_error_ttl: 60s
4272 + kubernetes_api:
4273 + listen:
4274 + address: localhost:8154
4275 + url_path_prefix: /
4276 + info_cache_ttl: 300s
4277 + info_cache_error_ttl: 60s
4278 +gitlab:
4279 + address: http://praefect.test
4280 + external_url: http://praefect.test
4281 + authentication_secret_file: /var/opt/gitlab/gitlab-kas/authentication_secret_file
4282 +observability:
4283 + listen:
4284 + address: localhost:8151
4285 + network: tcp
4286 + usage_reporting_period: 60s
4287 + logging:
4288 + level: info
4289 +redis:
4290 + network: unix
4291 + tls:
4292 + enabled: false
4293 + server:
4294 + address: '/var/opt/gitlab/redis/redis.socket'
4295 +api:
4296 + listen:
4297 + network: tcp
4298 + address: localhost:8153
4299 + authentication_secret_file: /var/opt/gitlab/gitlab-kas/authentication_secret_file
4300 +private_api:
4301 + listen:
4302 + network: tcp
4303 + address: localhost:8155
4304 + authentication_secret_file: /var/opt/gitlab/gitlab-kas/private_api_authentication_secret_file[2023-07-20T03:22:56+00:00] INFO: template[/var/opt/gitlab/gitlab-kas/gitlab-kas-config.yml] owner changed to 0
4305[2023-07-20T03:22:56+00:00] INFO: template[/var/opt/gitlab/gitlab-kas/gitlab-kas-config.yml] group changed to 998
4306[2023-07-20T03:22:56+00:00] INFO: template[/var/opt/gitlab/gitlab-kas/gitlab-kas-config.yml] mode changed to 640
4307 - change mode from '' to '0640'
4308 - change owner from '' to 'root'
4309 - change group from '' to 'git'
4310[2023-07-20T03:22:56+00:00] INFO: template[/var/opt/gitlab/gitlab-kas/gitlab-kas-config.yml] not queuing delayed action restart on runit_service[gitlab-kas] (delayed), as it's already been queued
4311 * env_dir[/opt/gitlab/etc/gitlab-kas/env] action create
4312 * directory[/opt/gitlab/etc/gitlab-kas/env] action create[2023-07-20T03:22:56+00:00] INFO: directory[/opt/gitlab/etc/gitlab-kas/env] created directory /opt/gitlab/etc/gitlab-kas/env
4313 - create new directory /opt/gitlab/etc/gitlab-kas/env
4314 * file[/opt/gitlab/etc/gitlab-kas/env/SSL_CERT_DIR] action create[2023-07-20T03:22:56+00:00] INFO: file[/opt/gitlab/etc/gitlab-kas/env/SSL_CERT_DIR] created file /opt/gitlab/etc/gitlab-kas/env/SSL_CERT_DIR
4315 - create new file /opt/gitlab/etc/gitlab-kas/env/SSL_CERT_DIR[2023-07-20T03:22:56+00:00] INFO: file[/opt/gitlab/etc/gitlab-kas/env/SSL_CERT_DIR] updated file contents /opt/gitlab/etc/gitlab-kas/env/SSL_CERT_DIR
4316 - update content in file /opt/gitlab/etc/gitlab-kas/env/SSL_CERT_DIR from none to 4f45cf
4317 - suppressed sensitive resource
4318 * file[/opt/gitlab/etc/gitlab-kas/env/OWN_PRIVATE_API_URL] action create[2023-07-20T03:22:56+00:00] INFO: file[/opt/gitlab/etc/gitlab-kas/env/OWN_PRIVATE_API_URL] created file /opt/gitlab/etc/gitlab-kas/env/OWN_PRIVATE_API_URL
4319 - create new file /opt/gitlab/etc/gitlab-kas/env/OWN_PRIVATE_API_URL[2023-07-20T03:22:56+00:00] INFO: file[/opt/gitlab/etc/gitlab-kas/env/OWN_PRIVATE_API_URL] updated file contents /opt/gitlab/etc/gitlab-kas/env/OWN_PRIVATE_API_URL
4320 - update content in file /opt/gitlab/etc/gitlab-kas/env/OWN_PRIVATE_API_URL from none to 45b898
4321 - suppressed sensitive resource
4322
4323 * service[gitlab-kas] action nothing (skipped due to action :nothing)
4324 * runit_service[gitlab-kas] action enable
4325 * ruby_block[restart_service] action nothing (skipped due to action :nothing)
4326 * ruby_block[restart_log_service] action nothing (skipped due to action :nothing)
4327 * ruby_block[reload_log_service] action nothing (skipped due to action :nothing)
4328 * directory[/opt/gitlab/sv/gitlab-kas] action create[2023-07-20T03:22:56+00:00] INFO: directory[/opt/gitlab/sv/gitlab-kas] created directory /opt/gitlab/sv/gitlab-kas
4329 - create new directory /opt/gitlab/sv/gitlab-kas[2023-07-20T03:22:56+00:00] INFO: directory[/opt/gitlab/sv/gitlab-kas] owner changed to 0
4330[2023-07-20T03:22:56+00:00] INFO: directory[/opt/gitlab/sv/gitlab-kas] group changed to 0
4331[2023-07-20T03:22:56+00:00] INFO: directory[/opt/gitlab/sv/gitlab-kas] mode changed to 755
4332 - change mode from '' to '0755'
4333 - change owner from '' to 'root'
4334 - change group from '' to 'root'
4335 * template[/opt/gitlab/sv/gitlab-kas/run] action create[2023-07-20T03:22:56+00:00] INFO: template[/opt/gitlab/sv/gitlab-kas/run] created file /opt/gitlab/sv/gitlab-kas/run
4336 - create new file /opt/gitlab/sv/gitlab-kas/run[2023-07-20T03:22:56+00:00] INFO: template[/opt/gitlab/sv/gitlab-kas/run] updated file contents /opt/gitlab/sv/gitlab-kas/run
4337 - update content in file /opt/gitlab/sv/gitlab-kas/run from none to ac037b
4338 --- /opt/gitlab/sv/gitlab-kas/run 2023-07-20 03:22:56.000000000 +0000
4339 +++ /opt/gitlab/sv/gitlab-kas/.chef-run20230720-37-b8wvo7 2023-07-20 03:22:56.000000000 +0000
4340 @@ -1,11 +1,22 @@
4341 +#!/bin/bash
4342 +
4343 +# Let runit capture all script error messages
4344 +exec 2>&1
4345 +
4346 +
4347 +
4348 +exec chpst -e /opt/gitlab/etc/gitlab-kas/env -P \
4349 + -u git:git \
4350 + -U git:git \
4351 + /opt/gitlab/embedded/bin/gitlab-kas --configuration-file /var/opt/gitlab/gitlab-kas/gitlab-kas-config.yml[2023-07-20T03:22:56+00:00] INFO: template[/opt/gitlab/sv/gitlab-kas/run] owner changed to 0
4352[2023-07-20T03:22:56+00:00] INFO: template[/opt/gitlab/sv/gitlab-kas/run] group changed to 0
4353[2023-07-20T03:22:56+00:00] INFO: template[/opt/gitlab/sv/gitlab-kas/run] mode changed to 755
4354 - change mode from '' to '0755'
4355 - change owner from '' to 'root'
4356 - change group from '' to 'root'
4357 * directory[/opt/gitlab/sv/gitlab-kas/log] action create[2023-07-20T03:22:56+00:00] INFO: directory[/opt/gitlab/sv/gitlab-kas/log] created directory /opt/gitlab/sv/gitlab-kas/log
4358 - create new directory /opt/gitlab/sv/gitlab-kas/log[2023-07-20T03:22:56+00:00] INFO: directory[/opt/gitlab/sv/gitlab-kas/log] owner changed to 0
4359[2023-07-20T03:22:56+00:00] INFO: directory[/opt/gitlab/sv/gitlab-kas/log] group changed to 0
4360[2023-07-20T03:22:56+00:00] INFO: directory[/opt/gitlab/sv/gitlab-kas/log] mode changed to 755
4361 - change mode from '' to '0755'
4362 - change owner from '' to 'root'
4363 - change group from '' to 'root'
4364 * directory[/opt/gitlab/sv/gitlab-kas/log/main] action create[2023-07-20T03:22:56+00:00] INFO: directory[/opt/gitlab/sv/gitlab-kas/log/main] created directory /opt/gitlab/sv/gitlab-kas/log/main
4365 - create new directory /opt/gitlab/sv/gitlab-kas/log/main[2023-07-20T03:22:56+00:00] INFO: directory[/opt/gitlab/sv/gitlab-kas/log/main] owner changed to 0
4366[2023-07-20T03:22:56+00:00] INFO: directory[/opt/gitlab/sv/gitlab-kas/log/main] group changed to 0
4367[2023-07-20T03:22:56+00:00] INFO: directory[/opt/gitlab/sv/gitlab-kas/log/main] mode changed to 755
4368 - change mode from '' to '0755'
4369 - change owner from '' to 'root'
4370 - change group from '' to 'root'
4371 * template[/opt/gitlab/sv/gitlab-kas/log/config] action create[2023-07-20T03:22:56+00:00] INFO: template[/opt/gitlab/sv/gitlab-kas/log/config] created file /opt/gitlab/sv/gitlab-kas/log/config
4372 - create new file /opt/gitlab/sv/gitlab-kas/log/config[2023-07-20T03:22:56+00:00] INFO: template[/opt/gitlab/sv/gitlab-kas/log/config] updated file contents /opt/gitlab/sv/gitlab-kas/log/config
4373 - update content in file /opt/gitlab/sv/gitlab-kas/log/config from none to 623c00
4374 --- /opt/gitlab/sv/gitlab-kas/log/config 2023-07-20 03:22:56.000000000 +0000
4375 +++ /opt/gitlab/sv/gitlab-kas/log/.chef-config20230720-37-g1ateh 2023-07-20 03:22:56.000000000 +0000
4376 @@ -1,6 +1,12 @@
4377 +s209715200
4378 +n30
4379 +t86400
4380 +!gzip
4381 +
4382 +[2023-07-20T03:22:56+00:00] INFO: template[/opt/gitlab/sv/gitlab-kas/log/config] owner changed to 0
4383[2023-07-20T03:22:56+00:00] INFO: template[/opt/gitlab/sv/gitlab-kas/log/config] group changed to 0
4384[2023-07-20T03:22:56+00:00] INFO: template[/opt/gitlab/sv/gitlab-kas/log/config] mode changed to 644
4385 - change mode from '' to '0644'
4386 - change owner from '' to 'root'
4387 - change group from '' to 'root'
4388[2023-07-20T03:22:56+00:00] INFO: template[/opt/gitlab/sv/gitlab-kas/log/config] sending create action to ruby_block[verify_chown_persisted_on_gitlab-kas] (immediate)
4389 * ruby_block[verify_chown_persisted_on_gitlab-kas] action create[2023-07-20T03:22:56+00:00] INFO: ruby_block[verify_chown_persisted_on_gitlab-kas] called
4390 - execute the ruby block verify_chown_persisted_on_gitlab-kas
4391 * ruby_block[verify_chown_persisted_on_gitlab-kas] action nothing (skipped due to action :nothing)
4392 * link[/var/log/gitlab/gitlab-kas/config] action create[2023-07-20T03:22:56+00:00] INFO: link[/var/log/gitlab/gitlab-kas/config] created
4393 - create symlink at /var/log/gitlab/gitlab-kas/config to /opt/gitlab/sv/gitlab-kas/log/config
4394 * template[/opt/gitlab/sv/gitlab-kas/log/run] action create[2023-07-20T03:22:56+00:00] INFO: template[/opt/gitlab/sv/gitlab-kas/log/run] created file /opt/gitlab/sv/gitlab-kas/log/run
4395 - create new file /opt/gitlab/sv/gitlab-kas/log/run[2023-07-20T03:22:56+00:00] INFO: template[/opt/gitlab/sv/gitlab-kas/log/run] updated file contents /opt/gitlab/sv/gitlab-kas/log/run
4396 - update content in file /opt/gitlab/sv/gitlab-kas/log/run from none to 22436c
4397 --- /opt/gitlab/sv/gitlab-kas/log/run 2023-07-20 03:22:56.000000000 +0000
4398 +++ /opt/gitlab/sv/gitlab-kas/log/.chef-run20230720-37-4fff7k 2023-07-20 03:22:56.000000000 +0000
4399 @@ -1,5 +1,10 @@
4400 +#!/bin/sh
4401 +exec chpst -P \
4402 + -U root:root \
4403 + -u root:root \
4404 + svlogd -tt /var/log/gitlab/gitlab-kas[2023-07-20T03:22:56+00:00] INFO: template[/opt/gitlab/sv/gitlab-kas/log/run] owner changed to 0
4405[2023-07-20T03:22:56+00:00] INFO: template[/opt/gitlab/sv/gitlab-kas/log/run] group changed to 0
4406[2023-07-20T03:22:56+00:00] INFO: template[/opt/gitlab/sv/gitlab-kas/log/run] mode changed to 755
4407 - change mode from '' to '0755'
4408 - change owner from '' to 'root'
4409 - change group from '' to 'root'
4410 * directory[/opt/gitlab/sv/gitlab-kas/env] action create[2023-07-20T03:22:56+00:00] INFO: directory[/opt/gitlab/sv/gitlab-kas/env] created directory /opt/gitlab/sv/gitlab-kas/env
4411 - create new directory /opt/gitlab/sv/gitlab-kas/env[2023-07-20T03:22:56+00:00] INFO: directory[/opt/gitlab/sv/gitlab-kas/env] owner changed to 0
4412[2023-07-20T03:22:56+00:00] INFO: directory[/opt/gitlab/sv/gitlab-kas/env] group changed to 0
4413[2023-07-20T03:22:56+00:00] INFO: directory[/opt/gitlab/sv/gitlab-kas/env] mode changed to 755
4414 - change mode from '' to '0755'
4415 - change owner from '' to 'root'
4416 - change group from '' to 'root'
4417 * ruby_block[Delete unmanaged env files for gitlab-kas service] action run (skipped due to only_if)
4418 * template[/opt/gitlab/sv/gitlab-kas/check] action create (skipped due to only_if)
4419 * template[/opt/gitlab/sv/gitlab-kas/finish] action create (skipped due to only_if)
4420 * directory[/opt/gitlab/sv/gitlab-kas/control] action create[2023-07-20T03:22:56+00:00] INFO: directory[/opt/gitlab/sv/gitlab-kas/control] created directory /opt/gitlab/sv/gitlab-kas/control
4421 - create new directory /opt/gitlab/sv/gitlab-kas/control[2023-07-20T03:22:56+00:00] INFO: directory[/opt/gitlab/sv/gitlab-kas/control] owner changed to 0
4422[2023-07-20T03:22:56+00:00] INFO: directory[/opt/gitlab/sv/gitlab-kas/control] group changed to 0
4423[2023-07-20T03:22:56+00:00] INFO: directory[/opt/gitlab/sv/gitlab-kas/control] mode changed to 755
4424 - change mode from '' to '0755'
4425 - change owner from '' to 'root'
4426 - change group from '' to 'root'
4427 * link[/opt/gitlab/init/gitlab-kas] action create[2023-07-20T03:22:56+00:00] INFO: link[/opt/gitlab/init/gitlab-kas] created
4428 - create symlink at /opt/gitlab/init/gitlab-kas to /opt/gitlab/embedded/bin/sv
4429 * file[/opt/gitlab/sv/gitlab-kas/down] action nothing (skipped due to action :nothing)
4430[2023-07-20T03:22:56+00:00] INFO: template[/opt/gitlab/sv/gitlab-kas/run] sending run action to ruby_block[restart_service] (delayed)
4431 * ruby_block[restart_service] action run (skipped due to only_if)
4432[2023-07-20T03:22:56+00:00] INFO: template[/opt/gitlab/sv/gitlab-kas/log/config] sending create action to ruby_block[reload_log_service] (delayed)
4433 * ruby_block[reload_log_service] action create
4434 * ruby_block[restart_service] action nothing (skipped due to action :nothing)
4435 * ruby_block[restart_log_service] action nothing (skipped due to action :nothing)
4436 * ruby_block[reload_log_service] action nothing (skipped due to action :nothing)
4437 * directory[/opt/gitlab/sv/gitlab-kas] action create (up to date)
4438 * template[/opt/gitlab/sv/gitlab-kas/run] action create (up to date)
4439 * directory[/opt/gitlab/sv/gitlab-kas/log] action create (up to date)
4440 * directory[/opt/gitlab/sv/gitlab-kas/log/main] action create (up to date)
4441 * template[/opt/gitlab/sv/gitlab-kas/log/config] action create (up to date)
4442 * ruby_block[verify_chown_persisted_on_gitlab-kas] action nothing (skipped due to action :nothing)
4443 * link[/var/log/gitlab/gitlab-kas/config] action create (up to date)
4444 * template[/opt/gitlab/sv/gitlab-kas/log/run] action create (up to date)
4445 * directory[/opt/gitlab/sv/gitlab-kas/env] action create (up to date)
4446 * ruby_block[Delete unmanaged env files for gitlab-kas service] action run (skipped due to only_if)
4447 * template[/opt/gitlab/sv/gitlab-kas/check] action create (skipped due to only_if)
4448 * template[/opt/gitlab/sv/gitlab-kas/finish] action create (skipped due to only_if)
4449 * directory[/opt/gitlab/sv/gitlab-kas/control] action create (up to date)
4450 * link[/opt/gitlab/init/gitlab-kas] action create (up to date)
4451 * file[/opt/gitlab/sv/gitlab-kas/down] action nothing (skipped due to action :nothing)
4452 * directory[/opt/gitlab/service] action create (up to date)
4453 * link[/opt/gitlab/service/gitlab-kas] action create[2023-07-20T03:22:56+00:00] INFO: link[/opt/gitlab/service/gitlab-kas] created
4454 - create symlink at /opt/gitlab/service/gitlab-kas to /opt/gitlab/sv/gitlab-kas
4455 * ruby_block[wait for gitlab-kas service socket] action run[2023-07-20T03:22:59+00:00] INFO: ruby_block[wait for gitlab-kas service socket] called
4456 - execute the ruby block wait for gitlab-kas service socket
4457 * file[/var/log/gitlab/gitlab-kas/current] action touch (skipped due to only_if)
4458[2023-07-20T03:22:59+00:00] INFO: ruby_block[reload_log_service] called
4459 - execute the ruby block reload_log_service
4460[2023-07-20T03:22:59+00:00] INFO: template[/opt/gitlab/sv/gitlab-kas/log/run] sending run action to ruby_block[restart_log_service] (delayed)
4461 * ruby_block[restart_log_service] action run
4462 * ruby_block[restart_service] action nothing (skipped due to action :nothing)
4463 * ruby_block[restart_log_service] action nothing (skipped due to action :nothing)
4464 * ruby_block[reload_log_service] action nothing (skipped due to action :nothing)
4465 * directory[/opt/gitlab/sv/gitlab-kas] action create (up to date)
4466 * template[/opt/gitlab/sv/gitlab-kas/run] action create (up to date)
4467 * directory[/opt/gitlab/sv/gitlab-kas/log] action create (up to date)
4468 * directory[/opt/gitlab/sv/gitlab-kas/log/main] action create (up to date)
4469 * template[/opt/gitlab/sv/gitlab-kas/log/config] action create (up to date)
4470 * ruby_block[verify_chown_persisted_on_gitlab-kas] action nothing (skipped due to action :nothing)
4471 * link[/var/log/gitlab/gitlab-kas/config] action create (up to date)
4472 * template[/opt/gitlab/sv/gitlab-kas/log/run] action create (up to date)
4473 * directory[/opt/gitlab/sv/gitlab-kas/env] action create (up to date)
4474 * ruby_block[Delete unmanaged env files for gitlab-kas service] action run (skipped due to only_if)
4475 * template[/opt/gitlab/sv/gitlab-kas/check] action create (skipped due to only_if)
4476 * template[/opt/gitlab/sv/gitlab-kas/finish] action create (skipped due to only_if)
4477 * directory[/opt/gitlab/sv/gitlab-kas/control] action create (up to date)
4478 * link[/opt/gitlab/init/gitlab-kas] action create (up to date)
4479 * file[/opt/gitlab/sv/gitlab-kas/down] action nothing (skipped due to action :nothing)
4480 * directory[/opt/gitlab/service] action create (up to date)
4481 * link[/opt/gitlab/service/gitlab-kas] action create (up to date)
4482 * ruby_block[wait for gitlab-kas service socket] action run (skipped due to not_if)
4483 * file[/var/log/gitlab/gitlab-kas/current] action touch (skipped due to only_if)
4484[2023-07-20T03:22:59+00:00] INFO: ruby_block[restart_log_service] called
4485 - execute the ruby block restart_log_service
4486 * directory[/opt/gitlab/service] action create (up to date)
4487 * link[/opt/gitlab/service/gitlab-kas] action create (up to date)
4488 * ruby_block[wait for gitlab-kas service socket] action run (skipped due to not_if)
4489 * file[/var/log/gitlab/gitlab-kas/current] action touch (skipped due to only_if)
4490
4491Recipe: gitlab::database_migrations
4492 * ruby_block[check remote PG version] action nothing (skipped due to action :nothing)
4493 * rails_migration[gitlab-rails] action run (skipped due to only_if)
4494Recipe: crond::disable
4495 * service[crond] action nothing (skipped due to action :nothing)
4496 * runit_service[crond] action disable
4497 * ruby_block[disable crond] action run (skipped due to only_if)
4498 (up to date)
4499Recipe: gitlab::puma_disable
4500 * service[puma] action nothing (skipped due to action :nothing)
4501 * runit_service[puma] action disable
4502 * ruby_block[disable puma] action run (skipped due to only_if)
4503 (up to date)
4504 * consul_service[rails] action delete
4505 * file[/var/opt/gitlab/consul/config.d/rails-service.json] action delete (up to date)
4506 (up to date)
4507Recipe: gitlab::sidekiq_disable
4508 * service[sidekiq] action nothing (skipped due to action :nothing)
4509 * runit_service[sidekiq] action disable
4510 * ruby_block[disable sidekiq] action run (skipped due to only_if)
4511 (up to date)
4512 * consul_service[sidekiq] action delete
4513 * file[/var/opt/gitlab/consul/config.d/sidekiq-service.json] action delete (up to date)
4514 (up to date)
4515Recipe: gitlab::gitlab-workhorse_disable
4516 * service[gitlab-workhorse] action nothing (skipped due to action :nothing)
4517 * runit_service[gitlab-workhorse] action disable
4518 * ruby_block[disable gitlab-workhorse] action run (skipped due to only_if)
4519 (up to date)
4520 * consul_service[workhorse] action delete
4521 * file[/var/opt/gitlab/consul/config.d/workhorse-service.json] action delete (up to date)
4522 (up to date)
4523Recipe: gitlab::mailroom_disable
4524 * service[mailroom] action nothing (skipped due to action :nothing)
4525 * runit_service[mailroom] action disable
4526 * ruby_block[disable mailroom] action run (skipped due to only_if)
4527 (up to date)
4528Recipe: gitlab::nginx_disable
4529 * service[nginx] action nothing (skipped due to action :nothing)
4530 * runit_service[nginx] action disable
4531 * ruby_block[disable nginx] action run (skipped due to only_if)
4532 (up to date)
4533 * consul_service[nginx] action delete
4534 * file[/var/opt/gitlab/consul/config.d/nginx-service.json] action delete (up to date)
4535 (up to date)
4536Recipe: gitlab::remote-syslog_disable
4537 * service[remote-syslog] action nothing (skipped due to action :nothing)
4538 * runit_service[remote-syslog] action disable
4539 * ruby_block[disable remote-syslog] action run (skipped due to only_if)
4540 (up to date)
4541Recipe: gitlab::bootstrap
4542 * file[/var/opt/gitlab/bootstrapped] action create[2023-07-20T03:22:59+00:00] INFO: file[/var/opt/gitlab/bootstrapped] created file /var/opt/gitlab/bootstrapped
4543 - create new file /var/opt/gitlab/bootstrapped[2023-07-20T03:22:59+00:00] INFO: file[/var/opt/gitlab/bootstrapped] updated file contents /var/opt/gitlab/bootstrapped
4544 - update content in file /var/opt/gitlab/bootstrapped from none to 4ae00c
4545 --- /var/opt/gitlab/bootstrapped 2023-07-20 03:22:59.000000000 +0000
4546 +++ /var/opt/gitlab/.chef-bootstrapped20230720-37-4smsyp 2023-07-20 03:22:59.000000000 +0000
4547 @@ -1 +1,2 @@
4548 +All your bootstraps are belong to Chef[2023-07-20T03:22:59+00:00] INFO: file[/var/opt/gitlab/bootstrapped] owner changed to 0
4549[2023-07-20T03:22:59+00:00] INFO: file[/var/opt/gitlab/bootstrapped] group changed to 0
4550[2023-07-20T03:22:59+00:00] INFO: file[/var/opt/gitlab/bootstrapped] mode changed to 600
4551 - change mode from '' to '0600'
4552 - change owner from '' to 'root'
4553 - change group from '' to 'root'
4554Recipe: gitlab::storage-check_disable
4555 * service[storage-check] action nothing (skipped due to action :nothing)
4556 * runit_service[storage-check] action disable
4557 * ruby_block[disable storage-check] action run (skipped due to only_if)
4558 (up to date)
4559Recipe: gitlab-pages::disable
4560 * service[gitlab-pages] action nothing (skipped due to action :nothing)
4561 * runit_service[gitlab-pages] action disable
4562 * ruby_block[disable gitlab-pages] action run (skipped due to only_if)
4563 (up to date)
4564Recipe: registry::disable
4565 * service[registry] action nothing (skipped due to action :nothing)
4566 * runit_service[registry] action disable
4567 * ruby_block[disable registry] action run (skipped due to only_if)
4568 (up to date)
4569Recipe: mattermost::disable
4570 * service[mattermost] action nothing (skipped due to action :nothing)
4571 * runit_service[mattermost] action disable
4572 * ruby_block[disable mattermost] action run (skipped due to only_if)
4573 (up to date)
4574Recipe: letsencrypt::disable
4575 * crond_job[letsencrypt-renew] action delete
4576 * file[/var/opt/gitlab/crond/letsencrypt-renew] action delete (up to date)
4577 (up to date)
4578Recipe: monitoring::node-exporter_disable
4579 * service[node-exporter] action nothing (skipped due to action :nothing)
4580 * runit_service[node-exporter] action disable
4581 * ruby_block[disable node-exporter] action run (skipped due to only_if)
4582 (up to date)
4583 * consul_service[node-exporter] action delete
4584 * file[/var/opt/gitlab/consul/config.d/node-exporter-service.json] action delete (up to date)
4585 (up to date)
4586Recipe: monitoring::gitlab-exporter_disable
4587 * service[gitlab-exporter] action nothing (skipped due to action :nothing)
4588 * runit_service[gitlab-exporter] action disable
4589 * ruby_block[disable gitlab-exporter] action run (skipped due to only_if)
4590 (up to date)
4591 * consul_service[gitlab-exporter] action delete
4592 * file[/var/opt/gitlab/consul/config.d/gitlab-exporter-service.json] action delete (up to date)
4593 (up to date)
4594Recipe: monitoring::redis-exporter_disable
4595 * service[redis-exporter] action nothing (skipped due to action :nothing)
4596 * runit_service[redis-exporter] action disable
4597 * ruby_block[disable redis-exporter] action run (skipped due to only_if)
4598 (up to date)
4599 * consul_service[redis-exporter] action delete
4600 * file[/var/opt/gitlab/consul/config.d/redis-exporer-service.json] action delete (up to date)
4601 (up to date)
4602Recipe: monitoring::user
4603 * account[Prometheus user and group] action create (up to date)
4604Recipe: monitoring::prometheus
4605 * directory[/var/opt/gitlab/prometheus] action create (up to date)
4606 * directory[/var/opt/gitlab/prometheus/rules] action create[2023-07-20T03:22:59+00:00] INFO: directory[/var/opt/gitlab/prometheus/rules] created directory /var/opt/gitlab/prometheus/rules
4607 - create new directory /var/opt/gitlab/prometheus/rules[2023-07-20T03:22:59+00:00] INFO: directory[/var/opt/gitlab/prometheus/rules] owner changed to 992
4608[2023-07-20T03:22:59+00:00] INFO: directory[/var/opt/gitlab/prometheus/rules] mode changed to 750
4609 - change mode from '' to '0750'
4610 - change owner from '' to 'gitlab-prometheus'
4611 * directory[/var/log/gitlab/prometheus] action create[2023-07-20T03:22:59+00:00] INFO: directory[/var/log/gitlab/prometheus] created directory /var/log/gitlab/prometheus
4612 - create new directory /var/log/gitlab/prometheus[2023-07-20T03:22:59+00:00] INFO: directory[/var/log/gitlab/prometheus] owner changed to 992
4613[2023-07-20T03:22:59+00:00] INFO: directory[/var/log/gitlab/prometheus] mode changed to 700
4614 - change mode from '' to '0700'
4615 - change owner from '' to 'gitlab-prometheus'
4616 * directory[/opt/gitlab/etc/prometheus/env] action create[2023-07-20T03:22:59+00:00] INFO: directory[/opt/gitlab/etc/prometheus/env] created directory /opt/gitlab/etc/prometheus/env
4617 - create new directory /opt/gitlab/etc/prometheus/env[2023-07-20T03:22:59+00:00] INFO: directory[/opt/gitlab/etc/prometheus/env] owner changed to 992
4618[2023-07-20T03:22:59+00:00] INFO: directory[/opt/gitlab/etc/prometheus/env] mode changed to 700
4619 - change mode from '' to '0700'
4620 - change owner from '' to 'gitlab-prometheus'
4621 * env_dir[/opt/gitlab/etc/prometheus/env] action create
4622 * directory[/opt/gitlab/etc/prometheus/env] action create (up to date)
4623 * file[/opt/gitlab/etc/prometheus/env/SSL_CERT_DIR] action create[2023-07-20T03:22:59+00:00] INFO: file[/opt/gitlab/etc/prometheus/env/SSL_CERT_DIR] created file /opt/gitlab/etc/prometheus/env/SSL_CERT_DIR
4624 - create new file /opt/gitlab/etc/prometheus/env/SSL_CERT_DIR[2023-07-20T03:22:59+00:00] INFO: file[/opt/gitlab/etc/prometheus/env/SSL_CERT_DIR] updated file contents /opt/gitlab/etc/prometheus/env/SSL_CERT_DIR
4625 - update content in file /opt/gitlab/etc/prometheus/env/SSL_CERT_DIR from none to 4f45cf
4626 - suppressed sensitive resource
4627
4628 * execute[reload prometheus] action nothing (skipped due to action :nothing)
4629 * file[Prometheus config] action create[2023-07-20T03:23:00+00:00] INFO: file[Prometheus config] created file /var/opt/gitlab/prometheus/prometheus.yml
4630 - create new file /var/opt/gitlab/prometheus/prometheus.yml[2023-07-20T03:23:00+00:00] INFO: file[Prometheus config] updated file contents /var/opt/gitlab/prometheus/prometheus.yml
4631 - update content in file /var/opt/gitlab/prometheus/prometheus.yml from none to d9f50f
4632 --- /var/opt/gitlab/prometheus/prometheus.yml 2023-07-20 03:23:00.000000000 +0000
4633 +++ /var/opt/gitlab/prometheus/.chef-prometheus20230720-37-5pfu25.yml 2023-07-20 03:23:00.000000000 +0000
4634 @@ -1,111 +1,222 @@
4635 +---
4636 +global:
4637 + scrape_interval: 15s
4638 + scrape_timeout: 15s
4639 + external_labels: {}
4640 +remote_read: []
4641 +remote_write: []
4642 +rule_files:
4643 +- "/var/opt/gitlab/prometheus/rules/*.rules"
4644 +scrape_configs:
4645 +- job_name: prometheus
4646 + static_configs:
4647 + - targets:
4648 + - localhost:9090
4649 +- job_name: kubernetes-cadvisor
4650 + scheme: https
4651 + tls_config:
4652 + ca_file: "/var/run/secrets/kubernetes.io/serviceaccount/ca.crt"
4653 + insecure_skip_verify: true
4654 + bearer_token_file: "/var/run/secrets/kubernetes.io/serviceaccount/token"
4655 + kubernetes_sd_configs:
4656 + - role: node
4657 + api_server: https://kubernetes.default.svc:443
4658 + tls_config:
4659 + ca_file: "/var/run/secrets/kubernetes.io/serviceaccount/ca.crt"
4660 + bearer_token_file: "/var/run/secrets/kubernetes.io/serviceaccount/token"
4661 + relabel_configs:
4662 + - action: labelmap
4663 + regex: __meta_kubernetes_node_label_(.+)
4664 + - target_label: __address__
4665 + replacement: kubernetes.default.svc:443
4666 + - source_labels:
4667 + - __meta_kubernetes_node_name
4668 + regex: "(.+)"
4669 + target_label: __metrics_path__
4670 + replacement: "/api/v1/nodes/${1}/proxy/metrics/cadvisor"
4671 + metric_relabel_configs:
4672 + - source_labels:
4673 + - pod_name
4674 + target_label: environment
4675 + regex: "(.+)-.+-.+"
4676 +- job_name: kubernetes-nodes
4677 + scheme: https
4678 + tls_config:
4679 + ca_file: "/var/run/secrets/kubernetes.io/serviceaccount/ca.crt"
4680 + insecure_skip_verify: true
4681 + bearer_token_file: "/var/run/secrets/kubernetes.io/serviceaccount/token"
4682 + kubernetes_sd_configs:
4683 + - role: node
4684 + api_server: https://kubernetes.default.svc:443
4685 + tls_config:
4686 + ca_file: "/var/run/secrets/kubernetes.io/serviceaccount/ca.crt"
4687 + bearer_token_file: "/var/run/secrets/kubernetes.io/serviceaccount/token"
4688 + relabel_configs:
4689 + - action: labelmap
4690 + regex: __meta_kubernetes_node_label_(.+)
4691 + - target_label: __address__
4692 + replacement: kubernetes.default.svc:443
4693 + - source_labels:
4694 + - __meta_kubernetes_node_name
4695 + regex: "(.+)"
4696 + target_label: __metrics_path__
4697 + replacement: "/api/v1/nodes/${1}/proxy/metrics"
4698 + metric_relabel_configs:
4699 + - source_labels:
4700 + - pod_name
4701 + target_label: environment
4702 + regex: "(.+)-.+-.+"
4703 +- job_name: kubernetes-pods
4704 + tls_config:
4705 + ca_file: "/var/run/secrets/kubernetes.io/serviceaccount/ca.crt"
4706 + insecure_skip_verify: true
4707 + bearer_token_file: "/var/run/secrets/kubernetes.io/serviceaccount/token"
4708 + kubernetes_sd_configs:
4709 + - role: pod
4710 + api_server: https://kubernetes.default.svc:443
4711 + tls_config:
4712 + ca_file: "/var/run/secrets/kubernetes.io/serviceaccount/ca.crt"
4713 + bearer_token_file: "/var/run/secrets/kubernetes.io/serviceaccount/token"
4714 + relabel_configs:
4715 + - source_labels:
4716 + - __meta_kubernetes_pod_annotation_prometheus_io_scrape
4717 + action: keep
4718 + regex: 'true'
4719 + - source_labels:
4720 + - __meta_kubernetes_pod_annotation_prometheus_io_path
4721 + action: replace
4722 + target_label: __metrics_path__
4723 + regex: "(.+)"
4724 + - source_labels:
4725 + - __address__
4726 + - __meta_kubernetes_pod_annotation_prometheus_io_port
4727 + action: replace
4728 + regex: "([^:]+)(?::[0-9]+)?;([0-9]+)"
4729 + replacement: "$1:$2"
4730 + target_label: __address__
4731 + - action: labelmap
4732 + regex: __meta_kubernetes_pod_label_(.+)
4733 + - source_labels:
4734 + - __meta_kubernetes_namespace
4735 + action: replace
4736 + target_label: kubernetes_namespace
4737 + - source_labels:
4738 + - __meta_kubernetes_pod_name
4739 + action: replace
4740 + target_label: kubernetes_pod_name
4741 +alerting:
4742 + alertmanagers:
4743 + - static_configs:
4744 + - targets:
4745 + - localhost:9093[2023-07-20T03:23:00+00:00] INFO: file[Prometheus config] owner changed to 992
4746[2023-07-20T03:23:00+00:00] INFO: file[Prometheus config] mode changed to 644
4747 - change mode from '' to '0644'
4748 - change owner from '' to 'gitlab-prometheus'
4749 * service[prometheus] action nothing (skipped due to action :nothing)
4750 * runit_service[prometheus] action enable
4751 * ruby_block[restart_service] action nothing (skipped due to action :nothing)
4752 * ruby_block[restart_log_service] action nothing (skipped due to action :nothing)
4753 * ruby_block[reload_log_service] action nothing (skipped due to action :nothing)
4754 * directory[/opt/gitlab/sv/prometheus] action create[2023-07-20T03:23:00+00:00] INFO: directory[/opt/gitlab/sv/prometheus] created directory /opt/gitlab/sv/prometheus
4755 - create new directory /opt/gitlab/sv/prometheus[2023-07-20T03:23:00+00:00] INFO: directory[/opt/gitlab/sv/prometheus] owner changed to 0
4756[2023-07-20T03:23:00+00:00] INFO: directory[/opt/gitlab/sv/prometheus] group changed to 0
4757[2023-07-20T03:23:00+00:00] INFO: directory[/opt/gitlab/sv/prometheus] mode changed to 755
4758 - change mode from '' to '0755'
4759 - change owner from '' to 'root'
4760 - change group from '' to 'root'
4761 * template[/opt/gitlab/sv/prometheus/run] action create[2023-07-20T03:23:00+00:00] INFO: template[/opt/gitlab/sv/prometheus/run] created file /opt/gitlab/sv/prometheus/run
4762 - create new file /opt/gitlab/sv/prometheus/run[2023-07-20T03:23:00+00:00] INFO: template[/opt/gitlab/sv/prometheus/run] updated file contents /opt/gitlab/sv/prometheus/run
4763 - update content in file /opt/gitlab/sv/prometheus/run from none to 60f91b
4764 --- /opt/gitlab/sv/prometheus/run 2023-07-20 03:23:00.000000000 +0000
4765 +++ /opt/gitlab/sv/prometheus/.chef-run20230720-37-ceb5ib 2023-07-20 03:23:00.000000000 +0000
4766 @@ -1,8 +1,16 @@
4767 +#!/bin/sh
4768 +exec 2>&1
4769 +
4770 +umask 077
4771 +exec chpst -P -e /opt/gitlab/etc/prometheus/env \
4772 + -U gitlab-prometheus:gitlab-prometheus \
4773 + -u gitlab-prometheus:gitlab-prometheus \
4774 + /opt/gitlab/embedded/bin/prometheus --web.listen-address=localhost:9090 --storage.tsdb.path=/var/opt/gitlab/prometheus/data --config.file=/var/opt/gitlab/prometheus/prometheus.yml[2023-07-20T03:23:00+00:00] INFO: template[/opt/gitlab/sv/prometheus/run] owner changed to 0
4775[2023-07-20T03:23:00+00:00] INFO: template[/opt/gitlab/sv/prometheus/run] group changed to 0
4776[2023-07-20T03:23:00+00:00] INFO: template[/opt/gitlab/sv/prometheus/run] mode changed to 755
4777 - change mode from '' to '0755'
4778 - change owner from '' to 'root'
4779 - change group from '' to 'root'
4780 * directory[/opt/gitlab/sv/prometheus/log] action create[2023-07-20T03:23:00+00:00] INFO: directory[/opt/gitlab/sv/prometheus/log] created directory /opt/gitlab/sv/prometheus/log
4781 - create new directory /opt/gitlab/sv/prometheus/log[2023-07-20T03:23:00+00:00] INFO: directory[/opt/gitlab/sv/prometheus/log] owner changed to 0
4782[2023-07-20T03:23:00+00:00] INFO: directory[/opt/gitlab/sv/prometheus/log] group changed to 0
4783[2023-07-20T03:23:00+00:00] INFO: directory[/opt/gitlab/sv/prometheus/log] mode changed to 755
4784 - change mode from '' to '0755'
4785 - change owner from '' to 'root'
4786 - change group from '' to 'root'
4787 * directory[/opt/gitlab/sv/prometheus/log/main] action create[2023-07-20T03:23:00+00:00] INFO: directory[/opt/gitlab/sv/prometheus/log/main] created directory /opt/gitlab/sv/prometheus/log/main
4788 - create new directory /opt/gitlab/sv/prometheus/log/main[2023-07-20T03:23:00+00:00] INFO: directory[/opt/gitlab/sv/prometheus/log/main] owner changed to 0
4789[2023-07-20T03:23:00+00:00] INFO: directory[/opt/gitlab/sv/prometheus/log/main] group changed to 0
4790[2023-07-20T03:23:00+00:00] INFO: directory[/opt/gitlab/sv/prometheus/log/main] mode changed to 755
4791 - change mode from '' to '0755'
4792 - change owner from '' to 'root'
4793 - change group from '' to 'root'
4794 * template[/opt/gitlab/sv/prometheus/log/config] action create[2023-07-20T03:23:00+00:00] INFO: template[/opt/gitlab/sv/prometheus/log/config] created file /opt/gitlab/sv/prometheus/log/config
4795 - create new file /opt/gitlab/sv/prometheus/log/config[2023-07-20T03:23:00+00:00] INFO: template[/opt/gitlab/sv/prometheus/log/config] updated file contents /opt/gitlab/sv/prometheus/log/config
4796 - update content in file /opt/gitlab/sv/prometheus/log/config from none to 623c00
4797 --- /opt/gitlab/sv/prometheus/log/config 2023-07-20 03:23:00.000000000 +0000
4798 +++ /opt/gitlab/sv/prometheus/log/.chef-config20230720-37-itelvu 2023-07-20 03:23:00.000000000 +0000
4799 @@ -1,6 +1,12 @@
4800 +s209715200
4801 +n30
4802 +t86400
4803 +!gzip
4804 +
4805 +[2023-07-20T03:23:00+00:00] INFO: template[/opt/gitlab/sv/prometheus/log/config] owner changed to 0
4806[2023-07-20T03:23:00+00:00] INFO: template[/opt/gitlab/sv/prometheus/log/config] group changed to 0
4807[2023-07-20T03:23:00+00:00] INFO: template[/opt/gitlab/sv/prometheus/log/config] mode changed to 644
4808 - change mode from '' to '0644'
4809 - change owner from '' to 'root'
4810 - change group from '' to 'root'
4811[2023-07-20T03:23:00+00:00] INFO: template[/opt/gitlab/sv/prometheus/log/config] sending create action to ruby_block[verify_chown_persisted_on_prometheus] (immediate)
4812 * ruby_block[verify_chown_persisted_on_prometheus] action create[2023-07-20T03:23:00+00:00] INFO: ruby_block[verify_chown_persisted_on_prometheus] called
4813 - execute the ruby block verify_chown_persisted_on_prometheus
4814 * ruby_block[verify_chown_persisted_on_prometheus] action nothing (skipped due to action :nothing)
4815 * link[/var/log/gitlab/prometheus/config] action create[2023-07-20T03:23:00+00:00] INFO: link[/var/log/gitlab/prometheus/config] created
4816 - create symlink at /var/log/gitlab/prometheus/config to /opt/gitlab/sv/prometheus/log/config
4817 * template[/opt/gitlab/sv/prometheus/log/run] action create[2023-07-20T03:23:00+00:00] INFO: template[/opt/gitlab/sv/prometheus/log/run] created file /opt/gitlab/sv/prometheus/log/run
4818 - create new file /opt/gitlab/sv/prometheus/log/run[2023-07-20T03:23:00+00:00] INFO: template[/opt/gitlab/sv/prometheus/log/run] updated file contents /opt/gitlab/sv/prometheus/log/run
4819 - update content in file /opt/gitlab/sv/prometheus/log/run from none to 8afdb7
4820 --- /opt/gitlab/sv/prometheus/log/run 2023-07-20 03:23:00.000000000 +0000
4821 +++ /opt/gitlab/sv/prometheus/log/.chef-run20230720-37-5wy0dr 2023-07-20 03:23:00.000000000 +0000
4822 @@ -1,5 +1,10 @@
4823 +#!/bin/sh
4824 +exec chpst -P \
4825 + -U root:root \
4826 + -u root:root \
4827 + svlogd -tt /var/log/gitlab/prometheus[2023-07-20T03:23:00+00:00] INFO: template[/opt/gitlab/sv/prometheus/log/run] owner changed to 0
4828[2023-07-20T03:23:00+00:00] INFO: template[/opt/gitlab/sv/prometheus/log/run] group changed to 0
4829[2023-07-20T03:23:00+00:00] INFO: template[/opt/gitlab/sv/prometheus/log/run] mode changed to 755
4830 - change mode from '' to '0755'
4831 - change owner from '' to 'root'
4832 - change group from '' to 'root'
4833 * directory[/opt/gitlab/sv/prometheus/env] action create[2023-07-20T03:23:00+00:00] INFO: directory[/opt/gitlab/sv/prometheus/env] created directory /opt/gitlab/sv/prometheus/env
4834 - create new directory /opt/gitlab/sv/prometheus/env[2023-07-20T03:23:00+00:00] INFO: directory[/opt/gitlab/sv/prometheus/env] owner changed to 0
4835[2023-07-20T03:23:00+00:00] INFO: directory[/opt/gitlab/sv/prometheus/env] group changed to 0
4836[2023-07-20T03:23:00+00:00] INFO: directory[/opt/gitlab/sv/prometheus/env] mode changed to 755
4837 - change mode from '' to '0755'
4838 - change owner from '' to 'root'
4839 - change group from '' to 'root'
4840 * ruby_block[Delete unmanaged env files for prometheus service] action run (skipped due to only_if)
4841 * template[/opt/gitlab/sv/prometheus/check] action create (skipped due to only_if)
4842 * template[/opt/gitlab/sv/prometheus/finish] action create (skipped due to only_if)
4843 * directory[/opt/gitlab/sv/prometheus/control] action create[2023-07-20T03:23:00+00:00] INFO: directory[/opt/gitlab/sv/prometheus/control] created directory /opt/gitlab/sv/prometheus/control
4844 - create new directory /opt/gitlab/sv/prometheus/control[2023-07-20T03:23:00+00:00] INFO: directory[/opt/gitlab/sv/prometheus/control] owner changed to 0
4845[2023-07-20T03:23:00+00:00] INFO: directory[/opt/gitlab/sv/prometheus/control] group changed to 0
4846[2023-07-20T03:23:00+00:00] INFO: directory[/opt/gitlab/sv/prometheus/control] mode changed to 755
4847 - change mode from '' to '0755'
4848 - change owner from '' to 'root'
4849 - change group from '' to 'root'
4850 * link[/opt/gitlab/init/prometheus] action create[2023-07-20T03:23:00+00:00] INFO: link[/opt/gitlab/init/prometheus] created
4851 - create symlink at /opt/gitlab/init/prometheus to /opt/gitlab/embedded/bin/sv
4852 * file[/opt/gitlab/sv/prometheus/down] action nothing (skipped due to action :nothing)
4853[2023-07-20T03:23:00+00:00] INFO: template[/opt/gitlab/sv/prometheus/run] sending run action to ruby_block[restart_service] (delayed)
4854 * ruby_block[restart_service] action run (skipped due to only_if)
4855[2023-07-20T03:23:00+00:00] INFO: template[/opt/gitlab/sv/prometheus/log/config] sending create action to ruby_block[reload_log_service] (delayed)
4856 * ruby_block[reload_log_service] action create
4857 * ruby_block[restart_service] action nothing (skipped due to action :nothing)
4858 * ruby_block[restart_log_service] action nothing (skipped due to action :nothing)
4859 * ruby_block[reload_log_service] action nothing (skipped due to action :nothing)
4860 * directory[/opt/gitlab/sv/prometheus] action create (up to date)
4861 * template[/opt/gitlab/sv/prometheus/run] action create (up to date)
4862 * directory[/opt/gitlab/sv/prometheus/log] action create (up to date)
4863 * directory[/opt/gitlab/sv/prometheus/log/main] action create (up to date)
4864 * template[/opt/gitlab/sv/prometheus/log/config] action create (up to date)
4865 * ruby_block[verify_chown_persisted_on_prometheus] action nothing (skipped due to action :nothing)
4866 * link[/var/log/gitlab/prometheus/config] action create (up to date)
4867 * template[/opt/gitlab/sv/prometheus/log/run] action create (up to date)
4868 * directory[/opt/gitlab/sv/prometheus/env] action create (up to date)
4869 * ruby_block[Delete unmanaged env files for prometheus service] action run (skipped due to only_if)
4870 * template[/opt/gitlab/sv/prometheus/check] action create (skipped due to only_if)
4871 * template[/opt/gitlab/sv/prometheus/finish] action create (skipped due to only_if)
4872 * directory[/opt/gitlab/sv/prometheus/control] action create (up to date)
4873 * link[/opt/gitlab/init/prometheus] action create (up to date)
4874 * file[/opt/gitlab/sv/prometheus/down] action nothing (skipped due to action :nothing)
4875 * directory[/opt/gitlab/service] action create (up to date)
4876 * link[/opt/gitlab/service/prometheus] action create[2023-07-20T03:23:00+00:00] INFO: link[/opt/gitlab/service/prometheus] created
4877 - create symlink at /opt/gitlab/service/prometheus to /opt/gitlab/sv/prometheus
4878 * ruby_block[wait for prometheus service socket] action run[2023-07-20T03:23:05+00:00] INFO: ruby_block[wait for prometheus service socket] called
4879 - execute the ruby block wait for prometheus service socket
4880 * file[/var/log/gitlab/prometheus/current] action touch (skipped due to only_if)
4881[2023-07-20T03:23:06+00:00] INFO: ruby_block[reload_log_service] called
4882 - execute the ruby block reload_log_service
4883[2023-07-20T03:23:06+00:00] INFO: template[/opt/gitlab/sv/prometheus/log/run] sending run action to ruby_block[restart_log_service] (delayed)
4884 * ruby_block[restart_log_service] action run
4885 * ruby_block[restart_service] action nothing (skipped due to action :nothing)
4886 * ruby_block[restart_log_service] action nothing (skipped due to action :nothing)
4887 * ruby_block[reload_log_service] action nothing (skipped due to action :nothing)
4888 * directory[/opt/gitlab/sv/prometheus] action create (up to date)
4889 * template[/opt/gitlab/sv/prometheus/run] action create (up to date)
4890 * directory[/opt/gitlab/sv/prometheus/log] action create (up to date)
4891 * directory[/opt/gitlab/sv/prometheus/log/main] action create (up to date)
4892 * template[/opt/gitlab/sv/prometheus/log/config] action create (up to date)
4893 * ruby_block[verify_chown_persisted_on_prometheus] action nothing (skipped due to action :nothing)
4894 * link[/var/log/gitlab/prometheus/config] action create (up to date)
4895 * template[/opt/gitlab/sv/prometheus/log/run] action create (up to date)
4896 * directory[/opt/gitlab/sv/prometheus/env] action create (up to date)
4897 * ruby_block[Delete unmanaged env files for prometheus service] action run (skipped due to only_if)
4898 * template[/opt/gitlab/sv/prometheus/check] action create (skipped due to only_if)
4899 * template[/opt/gitlab/sv/prometheus/finish] action create (skipped due to only_if)
4900 * directory[/opt/gitlab/sv/prometheus/control] action create (up to date)
4901 * link[/opt/gitlab/init/prometheus] action create (up to date)
4902 * file[/opt/gitlab/sv/prometheus/down] action nothing (skipped due to action :nothing)
4903 * directory[/opt/gitlab/service] action create (up to date)
4904 * link[/opt/gitlab/service/prometheus] action create (up to date)
4905 * ruby_block[wait for prometheus service socket] action run (skipped due to not_if)
4906 * file[/var/log/gitlab/prometheus/current] action touch (skipped due to only_if)
4907[2023-07-20T03:23:06+00:00] INFO: ruby_block[restart_log_service] called
4908 - execute the ruby block restart_log_service
4909 * directory[/opt/gitlab/service] action create (up to date)
4910 * link[/opt/gitlab/service/prometheus] action create (up to date)
4911 * ruby_block[wait for prometheus service socket] action run (skipped due to not_if)
4912 * file[/var/log/gitlab/prometheus/current] action touch (skipped due to only_if)
4913
4914 * consul_service[prometheus] action delete
4915 * file[/var/opt/gitlab/consul/config.d/prometheus-service.json] action delete (up to date)
4916 (up to date)
4917 * execute[/opt/gitlab/bin/gitlab-ctl start prometheus] action run
4918 [execute] ok: run: prometheus: (pid 684) 3s
4919[2023-07-20T03:23:07+00:00] INFO: execute[/opt/gitlab/bin/gitlab-ctl start prometheus] ran successfully
4920 - execute /opt/gitlab/bin/gitlab-ctl start prometheus
4921 * template[/var/opt/gitlab/prometheus/rules/gitlab.rules] action create[2023-07-20T03:23:07+00:00] INFO: template[/var/opt/gitlab/prometheus/rules/gitlab.rules] created file /var/opt/gitlab/prometheus/rules/gitlab.rules
4922 - create new file /var/opt/gitlab/prometheus/rules/gitlab.rules[2023-07-20T03:23:07+00:00] INFO: template[/var/opt/gitlab/prometheus/rules/gitlab.rules] updated file contents /var/opt/gitlab/prometheus/rules/gitlab.rules
4923 - update content in file /var/opt/gitlab/prometheus/rules/gitlab.rules from none to a55956
4924 --- /var/opt/gitlab/prometheus/rules/gitlab.rules 2023-07-20 03:23:07.000000000 +0000
4925 +++ /var/opt/gitlab/prometheus/rules/.chef-gitlab20230720-37-awijxi.rules 2023-07-20 03:23:07.000000000 +0000
4926 @@ -1,492 +1,984 @@
4927 +---
4928 +groups:
4929 +- name: GitLab
4930 + rules:
4931 + - record: instance:puma_utilization:ratio
4932 + expr: >
4933 + sum by (instance) (
4934 + puma_active_connections
4935 + ) /
4936 + sum by (instance) (
4937 + puma_max_threads
4938 + )
4939 + - record: job_route_method_code:gitlab_workhorse_http_request_duration_seconds_count:rate5m
4940 + expr: >
4941 + sum by (job, route, method, code) (
4942 + rate(gitlab_workhorse_http_request_duration_seconds_count[5m])
4943 + )
4944 +
4945 + - alert: ServiceDown
4946 + expr: avg_over_time(up[5m]) * 100 < 50
4947 + annotations:
4948 + description: The service {{ $labels.job }} instance {{ $labels.instance }} is
4949 + not responding for more than 50% of the time for 5 minutes.
4950 + summary: The service {{ $labels.job }} is not responding
4951 + - alert: RedisDown
4952 + expr: avg_over_time(redis_up[5m]) * 100 < 50
4953 + annotations:
4954 + description: The Redis service {{ $labels.job }} instance {{ $labels.instance
4955 + }} is not responding for more than 50% of the time for 5 minutes.
4956 + summary: The Redis service {{ $labels.job }} is not responding
4957 + - alert: PostgresDown
4958 + expr: avg_over_time(pg_up[5m]) * 100 < 50
4959 + annotations:
4960 + description: The Postgres service {{ $labels.job }} instance {{ $labels.instance
4961 + }} is not responding for more than 50% of the time for 5 minutes.
4962 + summary: The Postgres service {{ $labels.job }} is not responding
4963 + - alert: PumaQueueing
4964 + expr: avg_over_time(puma_queued_connections[30m]) > 1
4965 + annotations:
4966 + description: Puma instance {{ $labels.instance }} is queueing requests with
4967 + an average of {{ $value | printf "%.1f" }} over the last 30 minutes.
4968 + summary: Puma is queueing requests
4969 + - alert: HighPumaUtilization
4970 + expr: instance:puma_utilization:ratio * 100 > 90
4971 + for: 60m
4972 + annotations:
4973 + description: Puma instance {{ $labels.instance }} has more than 90% thread utilization ({{ $value | printf "%.1f" }}%) over the last 60 minutes.
4974 + summary: Puma is has high utilization
4975 + - alert: SidekiqJobsQueuing
4976 + expr: sum by (name) (sidekiq_queue_size) > 0
4977 + for: 60m
4978 + annotations:
4979 + summary: Sidekiq has jobs queued
4980 + description: Sidekiq queue {{ $labels.name }} has {{ $value }} jobs queued for 60 minutes.
4981 + - alert: HighgRPCResourceExhaustedRate
4982 + expr: >
4983 + sum without (grpc_code, grpc_method, grpc_service, grpc_type) (
4984 + rate(grpc_server_handled_total{grpc_code="ResourceExhausted"}[5m])
4985 + ) /
4986 + sum without (grpc_code, grpc_method, grpc_service, grpc_type) (
4987 + rate(grpc_server_handled_total[5m])
4988 + ) * 100 > 1
4989 + for: 60m
4990 + annotations:
4991 + summary: High gRPC ResourceExhausted error rate
4992 + description: Job {{ $labels.job }} instance {{ $labels.instance }} gRPC is returning more than 1% ({{ $value | printf "%.1f" }}%) ResourceExhausted errors over the last 60 minutes.
4993 + - alert: PostgresDatabaseDeadlocks
4994 + expr: increase(pg_stat_database_deadlocks[5m]) > 0
4995 + annotations:
4996 + summary: Postgres database has deadlocks
4997 + description: Postgres database {{ $labels.instance }} had {{ $value | printf "%d" }} deadlocks in the last 5 minutes.
4998 + - alert: PostgresDatabaseDeadlockCancels
4999 + expr: increase(pg_stat_database_deadlocks[5m]) > 0
5000 + annotations:
5001 + summary: Postgres database has queries canceled due to deadlocks
5002 + description: Postgres database {{ $labels.instance }} had {{ $value | printf "%d" }} queries canceled due to deadlocks in the last 5 minutes.
5003 + # Low-traffic - < 10 QPS (600 RPM)
5004 + - alert: WorkhorseHighErrorRate
5005 + expr: >
5006 + (
5007 + sum without (job, code) (
5008 + job_route_method_code:gitlab_workhorse_http_request_duration_seconds_count:rate5m{code=~"5.."}
5009 + ) /
5010 + sum without (job,code) (
5011 + job_route_method_code:gitlab_workhorse_http_request_duration_seconds_count:rate5m
5012 + ) < 10
5013 + ) * 100 > 50
5014 + annotations:
5015 + summary: Workhorse has high error rates
5016 + description: Workhorse route {{ $labels.route }} method {{ $labels.method }} has more than 50% errors ({{ $value | printf "%.1f" }}%) for the last 60 minutes.
5017 + # High-traffic - >= 10 QPS (600 RPM)
5018 + - alert: WorkhorseHighErrorRate
5019 + expr: >
5020 + (
5021 + sum without (job, code) (
5022 + job_route_method_code:gitlab_workhorse_http_request_duration_seconds_count:rate5m{code=~"5.."}
5023 + ) /
5024 + sum without (job,code) (
5025 + job_route_method_code:gitlab_workhorse_http_request_duration_seconds_count:rate5m
5026 + ) > 10
5027 + ) * 100 > 10
5028 + annotations:
5029 + summary: Workhorse has high error rates
5030 + description: Workhorse route {{ $labels.route }} method {{ $labels.method }} has more than 10% errors ({{ $value | printf "%.1f" }}%) for the last 60 minutes.
5031 +
5032 +###
5033 +# These metrics are top-level GitLab Service Level Indicators (SLIs). They can
5034 +# be used to monitor the overall health of a GitLab instance.
5035 +- name: Service Level Indicators
5036 + interval: 30s
5037 + rules:
5038 + # Service availability.
5039 + - record: gitlab_sli:job:availability:ratio
5040 + expr: >
5041 + avg by (job) (
5042 + avg_over_time(up[30s])
5043 + )
5044 + # Rails worker/thread capacity.
5045 + - record: gitlab_sli:rails_active_connections:avg30s
5046 + expr: >
5047 + sum(avg_over_time(puma_active_connections[30s]))
5048 + - record: gitlab_sli:rails_queued_connections:avg30s
5049 + expr: >
5050 + sum(avg_over_time(puma_queued_connections[30s]))
5051 + - record: gitlab_sli:rails_active_connections:max30s
5052 + expr: >
5053 + sum(max_over_time(puma_active_connections[30s]))
5054 + - record: gitlab_sli:rails_queued_connections:max30s
5055 + expr: >
5056 + sum(max_over_time(puma_queued_connections[30s]))
5057 + - record: gitlab_sli:rails_workers:avg30s
5058 + expr: >
5059 + count(avg_over_time(ruby_memory_bytes[30s])) or
5060 + sum(avg_over_time(puma_max_threads[30s]))
5061 + # Redis CPU use.
5062 + - record: gitlab_sli:redis_cpu_seconds:rate1m
5063 + expr: >
5064 + (sum(rate(redis_used_cpu_sys[1m])) + sum(rate(redis_used_cpu_user[1m]))) or
5065 + (sum(rate(redis_cpu_sys_seconds_total[1m])) + sum(rate(redis_cpu_user_seconds_total[1m])))
5066 + # Workhorse traffic.
5067 + - record: gitlab_sli:code_method_route:workhorse_http_request_count:rate1m
5068 + expr: >
5069 + sum by (code,method,route) (
5070 + rate(gitlab_workhorse_http_request_duration_seconds_count[1m])
5071 + )
5072 + - record: gitlab_sli:code_method_route:workhorse_http_request_duration_seconds:rate1m
5073 + expr: >
5074 + sum by (code,method,route) (
5075 + rate(gitlab_workhorse_http_request_duration_seconds_sum[1m])
5076 + )
5077 +# SLI - Apdex
5078 +- name: Service Level Indicators - Apdex
5079 + interval: 1m
5080 + rules:
5081 + # Gitaly goserver
5082 + # Satisfied -> 0.5 seconds
5083 + # Acceptable -> 1 seconds
5084 + - record: gitlab_sli:gitlab_component_apdex:ratio
5085 + labels:
5086 + job: gitaly
5087 + expr: >
5088 + (
5089 + sum(rate(grpc_server_handling_seconds_bucket{job="gitaly",grpc_type="unary",le="0.5",grpc_method!~"GarbageCollect|Fsck|RepackFull|RepackIncremental|CommitLanguages|CreateRepositoryFromURL|UserRebase|UserSquash|CreateFork|UserUpdateBranch|FindRemoteRepository|UserCherryPick|FetchRemote|UserRevert|FindRemoteRootRef"}[1m]))
5090 + +
5091 + sum(rate(grpc_server_handling_seconds_bucket{job="gitaly",grpc_type="unary",le="1",grpc_method!~"GarbageCollect|Fsck|RepackFull|RepackIncremental|CommitLanguages|CreateRepositoryFromURL|UserRebase|UserSquash|CreateFork|UserUpdateBranch|FindRemoteRepository|UserCherryPick|FetchRemote|UserRevert|FindRemoteRootRef"}[1m]))
5092 + )
5093 + /
5094 + 2 / sum(rate(grpc_server_handling_seconds_count{job="gitaly",grpc_type="unary",grpc_method!~"GarbageCollect|Fsck|RepackFull|RepackIncremental|CommitLanguages|CreateRepositoryFromURL|UserRebase|UserSquash|CreateFork|UserUpdateBranch|FindRemoteRepository|UserCherryPick|FetchRemote|UserRevert|FindRemoteRootRef"}[1m]))
5095 + # Sidekiq TODO: https://gitlab.com/gitlab-org/gitlab-foss/issues/56752
5096 + # - record: gitlab_sli:gitlab_component_apdex:ratio
5097 + # labels:
5098 + # job: gitlab-sidekiq
5099 + # expr: >
5100 + # (
5101 + # sum(rate(sidekiq_jobs_completion_time_seconds_bucket{le="25"}[1m]))
5102 + # +
5103 + # sum(rate(sidekiq_jobs_completion_time_seconds_bucket{le="50"}[1m]))
5104 + # )
5105 + # / 2
5106 + # /
5107 + # sum(rate(sidekiq_jobs_completion_time_seconds_count[1m]))
5108 +
5109 + # Workhorse
5110 + # Satisfied -> 1 seconds
5111 + # Acceptable -> 10 seconds
5112 + - record: gitlab_sli:gitlab_component_apdex:ratio
5113 + labels:
5114 + job: gitlab-workhorse
5115 + expr: >
5116 + (
5117 + sum(rate(gitlab_workhorse_http_request_duration_seconds_bucket{le="1",route!="^/([^/]+/){1,}[^/]+/uploads\\z",route!="^/api/v4/jobs/request\\z"}[1m]))
5118 + +
5119 + sum(rate(gitlab_workhorse_http_request_duration_seconds_bucket{le="10",route!="^/([^/]+/){1,}[^/]+/uploads\\z",route!="^/api/v4/jobs/request\\z"}[1m]))
5120 + )
5121 + /
5122 + 2 / sum(rate(gitlab_workhorse_http_request_duration_seconds_count{route!="^/([^/]+/){1,}[^/]+/uploads\\z",route!="^/api/v4/jobs/request\\z"}[1m]))
5123 +
5124 +# SLI - Errors
5125 +- name: Service Level Indicators - Errors
5126 + interval: 1m
5127 + rules:
5128 + # PostgreSQL
5129 + - record: gitlab_sli:gitlab_component_ops:rate
5130 + labels:
5131 + job: postgres
5132 + expr: >
5133 + sum(rate(pg_stat_database_xact_commit[1m])) +
5134 + sum(rate(pg_stat_database_xact_rollback[1m]))
5135 + - record: gitlab_sli:gitlab_component_errors:rate
5136 + labels:
5137 + job: postgres
5138 + expr: sum(rate(pg_stat_database_xact_rollback[1m]))
5139 + # Rails (Puma)
5140 + - record: gitlab_sli:gitlab_component_ops:rate
5141 + labels:
5142 + job: gitlab-rails
5143 + expr: >
5144 + sum(
5145 + rate(http_request_duration_seconds_count{job="gitlab-rails"}[1m])
5146 + )
5147 + - record: gitlab_sli:gitlab_component_errors:rate
5148 + labels:
5149 + job: gitlab-rails
5150 + expr: >
5151 + sum(
5152 + rate(http_request_duration_seconds_count{job="gitlab-rails",status=~"5.."}[1m])
5153 + )
5154 + # Sidekiq TODO: https://gitlab.com/gitlab-org/gitlab-foss/issues/56752
5155 + # - record: gitlab_sli:gitlab_component_ops:rate
5156 + # labels:
5157 + # job: gitlab-sidekiq
5158 + # expr: >
5159 + # sum (
5160 + # rate(sidekiq_jobs_started_total{job="gitlab-sidekiq"}[1m])
5161 + # )
5162 + # - record: gitlab_sli:gitlab_component_errors:rate
5163 + # labels:
5164 + # job: gitlab-sidekiq
5165 + # expr: >
5166 + # sum (
5167 + # rate(sidekiq_jobs_failed_total{job="gitlab-sidekiq"}[1m])
5168 + # )
5169 + # Workhorse
5170 + - record: gitlab_sli:gitlab_component_ops:rate
5171 + labels:
5172 + job: gitlab-workhorse
5173 + expr: >
5174 + sum(
5175 + rate(gitlab_workhorse_http_requests_total{job="gitlab-workhorse"}[1m])
5176 + )
5177 + - record: gitlab_sli:gitlab_component_errors:rate
5178 + labels:
5179 + job: gitlab-workhorse
5180 + expr: >
5181 + sum(
5182 + rate(gitlab_workhorse_http_requests_total{job="gitlab-workhorse",code=~"5.."}[1m])
5183 + )
5184 + ###
5185 + # Calculate service error ratios
5186 + - record: gitlab_sli:gitlab_component_errors:ratio
5187 + expr: >
5188 + gitlab_sli:gitlab_component_errors:rate /
5189 + gitlab_sli:gitlab_component_ops:rate
5190 +
5191 +- name: GitLab Saturation Ratios
5192 + interval: 1m
5193 + rules:
5194 + # type: *, component: cpu
5195 + # this measures average CPU across all the cores for the entire fleet for the given service
5196 + - record: gitlab_sli:gitlab_component_saturation:ratio
5197 + labels:
5198 + component: 'cpu'
5199 + expr: >
5200 + avg(1 - rate(node_cpu_seconds_total{mode="idle"}[1m]))
5201 +
5202 + # type: *, component: single_node_cpu
5203 + # this measures the maximum cpu availability across all the codes on a single server
5204 + # this can be helpful if CPU is not even distributed across the fleet.
5205 + - record: gitlab_sli:gitlab_component_saturation:ratio
5206 + labels:
5207 + component: 'single_node_cpu'
5208 + expr: >
5209 + max(
5210 + avg(1 - rate(node_cpu_seconds_total{mode="idle"}[1m]))
5211 + )
5212 +
5213 + - record: gitlab_sli:gitlab_component_saturation:ratio
5214 + labels:
5215 + component: 'disk_space'
5216 + expr: >
5217 + max(
5218 + (
5219 + (
5220 + node_filesystem_size_bytes{fstype=~"ext.|xfs|nfs.?"}
5221 + -
5222 + node_filesystem_free_bytes{fstype=~"ext.|xfs|nfs.?"}
5223 + )
5224 + /
5225 + node_filesystem_size_bytes{fstype=~"ext.|xfs|nfs.?"}
5226 + )
5227 + )
5228 +
5229 + - record: gitlab_sli:gitlab_component_saturation:ratio
5230 + labels:
5231 + component: 'memory'
5232 + expr: max (instance:node_memory_utilization:ratio)
5233 +
5234 + - record: gitlab_sli:gitlab_component_saturation:ratio
5235 + labels:
5236 + component: 'single_threaded_cpu'
5237 + expr: >
5238 + clamp_max(
5239 + max (
5240 + (rate(redis_cpu_user_seconds_total[1m]) + rate(redis_cpu_sys_seconds_total[1m])) or
5241 + (rate(redis_used_cpu_user[1m]) + rate(redis_used_cpu_sys[1m]))
5242 + ),
5243 + 1
5244 + )
5245 +
5246 + - record: gitlab_sli:gitlab_component_saturation:ratio
5247 + labels:
5248 + component: 'connection_pool'
5249 + expr: >
5250 + clamp_max(
5251 + max(
5252 + max_over_time(pgbouncer_pools_server_active_connections{user="gitlab"}[1m]) /
5253 + (
5254 + (
5255 + pgbouncer_pools_server_idle_connections{user="gitlab"} +
5256 + pgbouncer_pools_server_active_connections{user="gitlab"} +
5257 + pgbouncer_pools_server_testing_connections{user="gitlab"} +
5258 + pgbouncer_pools_server_used_connections{user="gitlab"} +
5259 + pgbouncer_pools_server_login_connections{user="gitlab"}
5260 + )
5261 + > 0
5262 + )
5263 + ), 1)
5264 +
5265 + # type: postgres-delayed, postgres-archive, patroni
5266 + - record: gitlab_sli:gitlab_component_saturation:ratio
5267 + labels:
5268 + component: 'active_db_connections'
5269 + expr: >
5270 + clamp_max(
5271 + max(
5272 + sum without(state) (pg_stat_activity_count{datname="gitlabhq_production", state!="idle"})
5273 + /
5274 + (sum without(state) (pg_stat_activity_count{datname="gitlabhq_production"}) > 0)
5275 + ), 1)
5276 +
5277 + # type: redis, redis-cache, component: redis_clients
5278 + # Records the saturation of redis client connections against a redis fleet
5279 + - record: gitlab_sli:gitlab_component_saturation:ratio
5280 + labels:
5281 + component: 'redis_clients'
5282 + expr: >
5283 + max(
5284 + max_over_time(redis_connected_clients[1m])
5285 + /
5286 + redis_config_maxclients
5287 + )
5288 +
5289 + # Aggregate over all components within a service using max
5290 + - record: gitlab_sli:gitlab_service_saturation:ratio
5291 + expr: >
5292 + max by (component) (gitlab_sli:gitlab_component_saturation:ratio)
5293 +
5294 + # Calculate the "sapdex" - the saturation apdex for the metric. 1 < less then soft limit, 0.5 < hard limit, else 0
5295 + - record: gitlab_sli:gitlab_component_saturation:ratio:sapdex
5296 + expr: >
5297 + clamp_min(gitlab_sli:gitlab_component_saturation:ratio <= on(component) group_left slo:max:soft:gitlab_sli:gitlab_component_saturation:ratio, 1)
5298 + or
5299 + clamp_min(clamp_max(gitlab_sli:gitlab_component_saturation:ratio > on(component) group_left slo:max:soft:gitlab_sli:gitlab_component_saturation:ratio, 0.5), 0.5)
5300 + or
5301 + clamp_max(gitlab_sli:gitlab_component_saturation:ratio > on(component) group_left slo:max:hard:gitlab_sli:gitlab_component_saturation:ratio, 0)
5302 +
5303 + # Open file descriptors
5304 + - record: gitlab_sli:gitlab_component_saturation:ratio
5305 + labels:
5306 + component: 'open_fds'
5307 + expr: >
5308 + max(
5309 + max_over_time(process_open_fds[1m])
5310 + /
5311 + max_over_time(process_max_fds[1m])
5312 + )
5313 + - record: gitlab_sli:gitlab_component_saturation:ratio
5314 + labels:
5315 + component: 'open_ruby_fds'
5316 + expr: >
5317 + max(
5318 + max_over_time(ruby_file_descriptors[1m])
5319 + /
5320 + max_over_time(ruby_process_max_fds[1m])
5321 + )
5322 +
5323 +# Unlike other service metrics, we record the stats for each component independently
5324 +- name: GitLab Saturation Ratios Stats
5325 + interval: 5m
5326 + rules:
5327 + # Average values for each service, over a week
5328 + - record: gitlab_sli:gitlab_component_saturation:ratio:avg_over_time_1w
5329 + expr: >
5330 + avg_over_time(gitlab_sli:gitlab_component_saturation:ratio[1w])
5331 +
5332 + # Using linear week-on-week growth, what prediction to we have for 2w from now?
5333 + - record: gitlab_sli:gitlab_component_saturation:ratio:predict_linear_2w
5334 + expr: >
5335 + predict_linear(gitlab_sli:gitlab_component_saturation:ratio:avg_over_time_1w[1w], 86400 * 14)
5336 +
5337 + # Using linear week-on-week growth, what prediction to we have for 30d from now?
5338 + - record: gitlab_sli:gitlab_component_saturation:ratio:predict_linear_30d
5339 + expr: >
5340 + predict_linear(gitlab_sli:gitlab_component_saturation:ratio:avg_over_time_1w[1w], 86400 * 30)
5341 +
5342 + # Sapdex, average for week
5343 + - record: gitlab_sli:gitlab_component_saturation:ratio:sapdex:avg_over_time_1w
5344 + expr: >
5345 + avg_over_time(gitlab_sli:gitlab_component_saturation:ratio:sapdex[1w])
5346 +
5347 + # Sapdex long term trend forecasting
5348 + # Using linear week-on-week growth, what prediction to we have for 30d from now?
5349 + - record: gitlab_sli:gitlab_component_saturation:ratio:sapdex:avg_over_time_1w:predict_linear_30d
5350 + expr: >
5351 + predict_linear(gitlab_sli:gitlab_component_saturation:ratio:sapdex:avg_over_time_1w[1w], 86400 * 30)
5352 +
5353 +###
5354 +# Metrics used by Usage Ping reports. These are primarily for self-managed installations.
5355 +# See https://docs.gitlab.com/ee/development/telemetry/usage_ping.html#prometheus-queries
5356 +#
5357 +- name: GitLab Usage Ping
5358 + interval: 5m
5359 + rules:
5360 + ###
5361 + # Application wide metrics
5362 + #
5363 + - record: gitlab_usage_ping:ops:rate5m
5364 + labels:
5365 + service: workhorse
5366 + component: http_requests
5367 + expr: sum (rate (gitlab_workhorse_http_requests_total[5m]))
5368 +
5369 + # Database Primary Performance Indicator: Query apdex
5370 + # Target -> 100ms
5371 + # Tolerable -> 250ms
5372 + - record: gitlab_usage_ping:sql_duration_apdex:ratio_rate5m
5373 + expr: >
5374 + (
5375 + sum(rate(gitlab_sql_duration_seconds_bucket{le="0.1"}[5m]))
5376 + +
5377 + sum(rate(gitlab_sql_duration_seconds_bucket{le="0.25"}[5m]))
5378 + )
5379 + /
5380 + 2
5381 + /
5382 + (
5383 + sum(rate(gitlab_sql_duration_seconds_count[5m])) > 0
5384 + )
5385 +
5386 + - record: gitlab_usage_ping:gitaly_apdex:ratio_avg_over_time_5m
5387 + expr: avg_over_time(gitlab_sli:gitlab_component_apdex:ratio{job="gitaly"}[5m])
5388 +
5389 + ###
5390 + # Node level metrics
5391 + #
5392 + - record: gitlab_usage_ping:node_cpus:count
5393 + expr: max (instance:node_cpus:count) by (instance)
5394 + - record: gitlab_usage_ping:node_memory_total_bytes:max
5395 + expr: max (node_memory_MemTotal_bytes) by (instance)
5396 + - record: gitlab_usage_ping:node_cpu_utilization:avg
5397 + expr: avg (instance:node_cpu_utilization:ratio) by (instance)
5398 + - record: gitlab_usage_ping:node_memory_utilization:avg
5399 + expr: avg (instance:node_memory_utilization:ratio) by (instance)
5400 +
5401 + ###
5402 + # Service level metrics
5403 + #
5404 + - record: gitlab_usage_ping:node_service_process_resident_memory_bytes:avg
5405 + expr: avg by (instance, job) ({__name__ =~ "(ruby_){0,1}process_resident_memory_bytes", job != "gitlab_exporter_process"})
5406 + - record: gitlab_usage_ping:node_service_process_unique_memory_bytes:avg
5407 + expr: avg by (instance, job) ({__name__ =~ "(ruby_){0,1}process_unique_memory_bytes", job != "gitlab_exporter_process"})
5408 + - record: gitlab_usage_ping:node_service_process_proportional_memory_bytes:avg
5409 + expr: avg by (instance, job) ({__name__ =~ "(ruby_){0,1}process_proportional_memory_bytes", job != "gitlab_exporter_process"})
5410 + - record: gitlab_usage_ping:node_service_process:count
5411 + expr: >
5412 + count by (instance, job) (
5413 + {__name__ =~ "(ruby_){0,1}process_start_time_seconds", job != "gitlab_exporter_process"}
5414 + )
5415 + - record: gitlab_usage_ping:node_service_app_server_workers:sum
5416 + labels:
5417 + server: puma
5418 + expr: sum by (instance, job) (puma_workers)[2023-07-20T03:23:07+00:00] INFO: template[/var/opt/gitlab/prometheus/rules/gitlab.rules] owner changed to 992
5419[2023-07-20T03:23:07+00:00] INFO: template[/var/opt/gitlab/prometheus/rules/gitlab.rules] mode changed to 644
5420 - change mode from '' to '0644'
5421 - change owner from '' to 'gitlab-prometheus'
5422[2023-07-20T03:23:07+00:00] INFO: template[/var/opt/gitlab/prometheus/rules/gitlab.rules] not queuing delayed action run on execute[reload prometheus] (delayed), as it's already been queued
5423 * template[/var/opt/gitlab/prometheus/rules/node.rules] action create[2023-07-20T03:23:07+00:00] INFO: template[/var/opt/gitlab/prometheus/rules/node.rules] created file /var/opt/gitlab/prometheus/rules/node.rules
5424 - create new file /var/opt/gitlab/prometheus/rules/node.rules[2023-07-20T03:23:07+00:00] INFO: template[/var/opt/gitlab/prometheus/rules/node.rules] updated file contents /var/opt/gitlab/prometheus/rules/node.rules
5425 - update content in file /var/opt/gitlab/prometheus/rules/node.rules from none to bea78d
5426 --- /var/opt/gitlab/prometheus/rules/node.rules 2023-07-20 03:23:07.000000000 +0000
5427 +++ /var/opt/gitlab/prometheus/rules/.chef-node20230720-37-lti0h2.rules 2023-07-20 03:23:07.000000000 +0000
5428 @@ -1,42 +1,84 @@
5429 +groups:
5430 +- name: Node
5431 + rules:
5432 + - record: instance:node_cpus:count
5433 + expr: count without(cpu, mode) (node_cpu{mode="idle"})
5434 + - record: instance:node_cpus:count
5435 + expr: count without(cpu, mode) (node_cpu_seconds_total{mode="idle"})
5436 + - record: instance_cpu:node_cpu_seconds_not_idle:rate5m
5437 + expr: sum without(mode) (rate(node_cpu{mode!="idle"}[5m]))
5438 + - record: instance_cpu:node_cpu_seconds_not_idle:rate5m
5439 + expr: sum without(mode) (rate(node_cpu_seconds_total{mode!="idle"}[5m]))
5440 + - record: instance_mode:node_cpu_seconds:rate5m
5441 + expr: sum without(cpu) (rate(node_cpu[5m]))
5442 + - record: instance_mode:node_cpu_seconds:rate5m
5443 + expr: sum without(cpu) (rate(node_cpu_seconds_total[5m]))
5444 + - record: instance:node_cpu_utilization:ratio
5445 + expr: sum without(mode) (instance_mode:node_cpu_seconds:rate5m{mode!="idle"})
5446 + / instance:node_cpus:count
5447 + - record: instance:node_memory_utilization:ratio
5448 + expr: >
5449 + 1 - (
5450 + node_memory_MemAvailable_bytes or
5451 + (node_memory_MemFree_bytes + node_memory_Buffers_bytes + node_memory_Cached_bytes)
5452 + ) / node_memory_MemTotal_bytes
5453 + - record: instance:node_filesystem_avail:ratio
5454 + expr: node_filesystem_avail_bytes / (node_filesystem_size_bytes > 0)
5455 + - alert: FilesystemAlmostFull
5456 + expr: instance:node_filesystem_avail:ratio * 100 < 5 and node_filesystem_avail_bytes < 10 * 1024 * 1024 * 1024
5457 + for: 10m
5458 + annotations:
5459 + description: The filesystem {{ $labels.device }}:{{ $labels.mountpoint }} on
5460 + {{ $labels.instance }} has {{ $value | printf "%.2f" }}% space available.
5461 + summary: The filesystem {{ $labels.device }}:{{ $labels.mountpoint }} is almost
5462 + full
5463 + - alert: FilesystemFullIn1Day
5464 + expr: predict_linear(node_filesystem_avail_bytes[6h], 24 * 3600) < 0
5465 + for: 30m
5466 + annotations:
5467 + description: The filesystem {{ $labels.device }}:{{ $labels.mountpoint }} on
5468 + {{ $labels.instance }} will be full in the next 24 hours.
5469 + summary: The filesystem {{ $labels.device }}:{{ $labels.mountpoint }} will be
5470 + full within 24 hours[2023-07-20T03:23:07+00:00] INFO: template[/var/opt/gitlab/prometheus/rules/node.rules] owner changed to 992
5471[2023-07-20T03:23:07+00:00] INFO: template[/var/opt/gitlab/prometheus/rules/node.rules] mode changed to 644
5472 - change mode from '' to '0644'
5473 - change owner from '' to 'gitlab-prometheus'
5474[2023-07-20T03:23:07+00:00] INFO: template[/var/opt/gitlab/prometheus/rules/node.rules] not queuing delayed action run on execute[reload prometheus] (delayed), as it's already been queued
5475Recipe: monitoring::alertmanager
5476 * directory[/var/opt/gitlab/alertmanager] action create[2023-07-20T03:23:07+00:00] INFO: directory[/var/opt/gitlab/alertmanager] created directory /var/opt/gitlab/alertmanager
5477 - create new directory /var/opt/gitlab/alertmanager[2023-07-20T03:23:07+00:00] INFO: directory[/var/opt/gitlab/alertmanager] owner changed to 992
5478[2023-07-20T03:23:07+00:00] INFO: directory[/var/opt/gitlab/alertmanager] mode changed to 750
5479 - change mode from '' to '0750'
5480 - change owner from '' to 'gitlab-prometheus'
5481 * directory[/var/log/gitlab/alertmanager] action create[2023-07-20T03:23:07+00:00] INFO: directory[/var/log/gitlab/alertmanager] created directory /var/log/gitlab/alertmanager
5482 - create new directory /var/log/gitlab/alertmanager[2023-07-20T03:23:07+00:00] INFO: directory[/var/log/gitlab/alertmanager] owner changed to 992
5483[2023-07-20T03:23:07+00:00] INFO: directory[/var/log/gitlab/alertmanager] mode changed to 700
5484 - change mode from '' to '0700'
5485 - change owner from '' to 'gitlab-prometheus'
5486 * directory[/opt/gitlab/etc/alertmanager/env] action create[2023-07-20T03:23:07+00:00] INFO: directory[/opt/gitlab/etc/alertmanager/env] created directory /opt/gitlab/etc/alertmanager/env
5487 - create new directory /opt/gitlab/etc/alertmanager/env[2023-07-20T03:23:07+00:00] INFO: directory[/opt/gitlab/etc/alertmanager/env] owner changed to 992
5488[2023-07-20T03:23:07+00:00] INFO: directory[/opt/gitlab/etc/alertmanager/env] mode changed to 700
5489 - change mode from '' to '0700'
5490 - change owner from '' to 'gitlab-prometheus'
5491 * env_dir[/opt/gitlab/etc/alertmanager/env] action create
5492 * directory[/opt/gitlab/etc/alertmanager/env] action create (up to date)
5493 * file[/opt/gitlab/etc/alertmanager/env/SSL_CERT_DIR] action create[2023-07-20T03:23:07+00:00] INFO: file[/opt/gitlab/etc/alertmanager/env/SSL_CERT_DIR] created file /opt/gitlab/etc/alertmanager/env/SSL_CERT_DIR
5494 - create new file /opt/gitlab/etc/alertmanager/env/SSL_CERT_DIR[2023-07-20T03:23:07+00:00] INFO: file[/opt/gitlab/etc/alertmanager/env/SSL_CERT_DIR] updated file contents /opt/gitlab/etc/alertmanager/env/SSL_CERT_DIR
5495 - update content in file /opt/gitlab/etc/alertmanager/env/SSL_CERT_DIR from none to 4f45cf
5496 - suppressed sensitive resource
5497
5498 * file[Alertmanager config] action create[2023-07-20T03:23:07+00:00] INFO: file[Alertmanager config] created file /var/opt/gitlab/alertmanager/alertmanager.yml
5499 - create new file /var/opt/gitlab/alertmanager/alertmanager.yml[2023-07-20T03:23:07+00:00] INFO: file[Alertmanager config] updated file contents /var/opt/gitlab/alertmanager/alertmanager.yml
5500 - update content in file /var/opt/gitlab/alertmanager/alertmanager.yml from none to 21b7be
5501 --- /var/opt/gitlab/alertmanager/alertmanager.yml 2023-07-20 03:23:07.000000000 +0000
5502 +++ /var/opt/gitlab/alertmanager/.chef-alertmanager20230720-37-deq32d.yml 2023-07-20 03:23:07.000000000 +0000
5503 @@ -1,9 +1,18 @@
5504 +---
5505 +global: {}
5506 +templates: []
5507 +route:
5508 + receiver: default-receiver
5509 + routes: []
5510 +receivers:
5511 +- name: default-receiver
5512 +inhibit_rules: [][2023-07-20T03:23:07+00:00] INFO: file[Alertmanager config] owner changed to 992
5513[2023-07-20T03:23:07+00:00] INFO: file[Alertmanager config] mode changed to 644
5514 - change mode from '' to '0644'
5515 - change owner from '' to 'gitlab-prometheus'
5516[2023-07-20T03:23:07+00:00] INFO: file[Alertmanager config] not queuing delayed action restart on runit_service[alertmanager] (delayed), as it's already been queued
5517 * service[alertmanager] action nothing (skipped due to action :nothing)
5518 * runit_service[alertmanager] action enable
5519 * ruby_block[restart_service] action nothing (skipped due to action :nothing)
5520 * ruby_block[restart_log_service] action nothing (skipped due to action :nothing)
5521 * ruby_block[reload_log_service] action nothing (skipped due to action :nothing)
5522 * directory[/opt/gitlab/sv/alertmanager] action create[2023-07-20T03:23:07+00:00] INFO: directory[/opt/gitlab/sv/alertmanager] created directory /opt/gitlab/sv/alertmanager
5523 - create new directory /opt/gitlab/sv/alertmanager[2023-07-20T03:23:07+00:00] INFO: directory[/opt/gitlab/sv/alertmanager] owner changed to 0
5524[2023-07-20T03:23:07+00:00] INFO: directory[/opt/gitlab/sv/alertmanager] group changed to 0
5525[2023-07-20T03:23:07+00:00] INFO: directory[/opt/gitlab/sv/alertmanager] mode changed to 755
5526 - change mode from '' to '0755'
5527 - change owner from '' to 'root'
5528 - change group from '' to 'root'
5529 * template[/opt/gitlab/sv/alertmanager/run] action create[2023-07-20T03:23:07+00:00] INFO: template[/opt/gitlab/sv/alertmanager/run] created file /opt/gitlab/sv/alertmanager/run
5530 - create new file /opt/gitlab/sv/alertmanager/run[2023-07-20T03:23:07+00:00] INFO: template[/opt/gitlab/sv/alertmanager/run] updated file contents /opt/gitlab/sv/alertmanager/run
5531 - update content in file /opt/gitlab/sv/alertmanager/run from none to 36da8b
5532 --- /opt/gitlab/sv/alertmanager/run 2023-07-20 03:23:07.000000000 +0000
5533 +++ /opt/gitlab/sv/alertmanager/.chef-run20230720-37-bkdodj 2023-07-20 03:23:07.000000000 +0000
5534 @@ -1,8 +1,16 @@
5535 +#!/bin/sh
5536 +exec 2>&1
5537 +
5538 +umask 077
5539 +exec chpst -P -e /opt/gitlab/etc/alertmanager/env \
5540 + -U gitlab-prometheus:gitlab-prometheus \
5541 + -u gitlab-prometheus:gitlab-prometheus \
5542 + /opt/gitlab/embedded/bin/alertmanager --web.listen-address=localhost:9093 --storage.path=/var/opt/gitlab/alertmanager/data --config.file=/var/opt/gitlab/alertmanager/alertmanager.yml[2023-07-20T03:23:07+00:00] INFO: template[/opt/gitlab/sv/alertmanager/run] owner changed to 0
5543[2023-07-20T03:23:07+00:00] INFO: template[/opt/gitlab/sv/alertmanager/run] group changed to 0
5544[2023-07-20T03:23:07+00:00] INFO: template[/opt/gitlab/sv/alertmanager/run] mode changed to 755
5545 - change mode from '' to '0755'
5546 - change owner from '' to 'root'
5547 - change group from '' to 'root'
5548 * directory[/opt/gitlab/sv/alertmanager/log] action create[2023-07-20T03:23:07+00:00] INFO: directory[/opt/gitlab/sv/alertmanager/log] created directory /opt/gitlab/sv/alertmanager/log
5549 - create new directory /opt/gitlab/sv/alertmanager/log[2023-07-20T03:23:07+00:00] INFO: directory[/opt/gitlab/sv/alertmanager/log] owner changed to 0
5550[2023-07-20T03:23:07+00:00] INFO: directory[/opt/gitlab/sv/alertmanager/log] group changed to 0
5551[2023-07-20T03:23:07+00:00] INFO: directory[/opt/gitlab/sv/alertmanager/log] mode changed to 755
5552 - change mode from '' to '0755'
5553 - change owner from '' to 'root'
5554 - change group from '' to 'root'
5555 * directory[/opt/gitlab/sv/alertmanager/log/main] action create[2023-07-20T03:23:07+00:00] INFO: directory[/opt/gitlab/sv/alertmanager/log/main] created directory /opt/gitlab/sv/alertmanager/log/main
5556 - create new directory /opt/gitlab/sv/alertmanager/log/main[2023-07-20T03:23:07+00:00] INFO: directory[/opt/gitlab/sv/alertmanager/log/main] owner changed to 0
5557[2023-07-20T03:23:07+00:00] INFO: directory[/opt/gitlab/sv/alertmanager/log/main] group changed to 0
5558[2023-07-20T03:23:07+00:00] INFO: directory[/opt/gitlab/sv/alertmanager/log/main] mode changed to 755
5559 - change mode from '' to '0755'
5560 - change owner from '' to 'root'
5561 - change group from '' to 'root'
5562 * template[/opt/gitlab/sv/alertmanager/log/config] action create[2023-07-20T03:23:07+00:00] INFO: template[/opt/gitlab/sv/alertmanager/log/config] created file /opt/gitlab/sv/alertmanager/log/config
5563 - create new file /opt/gitlab/sv/alertmanager/log/config[2023-07-20T03:23:07+00:00] INFO: template[/opt/gitlab/sv/alertmanager/log/config] updated file contents /opt/gitlab/sv/alertmanager/log/config
5564 - update content in file /opt/gitlab/sv/alertmanager/log/config from none to 623c00
5565 --- /opt/gitlab/sv/alertmanager/log/config 2023-07-20 03:23:07.000000000 +0000
5566 +++ /opt/gitlab/sv/alertmanager/log/.chef-config20230720-37-i826er 2023-07-20 03:23:07.000000000 +0000
5567 @@ -1,6 +1,12 @@
5568 +s209715200
5569 +n30
5570 +t86400
5571 +!gzip
5572 +
5573 +[2023-07-20T03:23:07+00:00] INFO: template[/opt/gitlab/sv/alertmanager/log/config] owner changed to 0
5574[2023-07-20T03:23:07+00:00] INFO: template[/opt/gitlab/sv/alertmanager/log/config] group changed to 0
5575[2023-07-20T03:23:07+00:00] INFO: template[/opt/gitlab/sv/alertmanager/log/config] mode changed to 644
5576 - change mode from '' to '0644'
5577 - change owner from '' to 'root'
5578 - change group from '' to 'root'
5579[2023-07-20T03:23:07+00:00] INFO: template[/opt/gitlab/sv/alertmanager/log/config] sending create action to ruby_block[verify_chown_persisted_on_alertmanager] (immediate)
5580 * ruby_block[verify_chown_persisted_on_alertmanager] action create[2023-07-20T03:23:07+00:00] INFO: ruby_block[verify_chown_persisted_on_alertmanager] called
5581 - execute the ruby block verify_chown_persisted_on_alertmanager
5582 * ruby_block[verify_chown_persisted_on_alertmanager] action nothing (skipped due to action :nothing)
5583 * link[/var/log/gitlab/alertmanager/config] action create[2023-07-20T03:23:07+00:00] INFO: link[/var/log/gitlab/alertmanager/config] created
5584 - create symlink at /var/log/gitlab/alertmanager/config to /opt/gitlab/sv/alertmanager/log/config
5585 * template[/opt/gitlab/sv/alertmanager/log/run] action create[2023-07-20T03:23:07+00:00] INFO: template[/opt/gitlab/sv/alertmanager/log/run] created file /opt/gitlab/sv/alertmanager/log/run
5586 - create new file /opt/gitlab/sv/alertmanager/log/run[2023-07-20T03:23:07+00:00] INFO: template[/opt/gitlab/sv/alertmanager/log/run] updated file contents /opt/gitlab/sv/alertmanager/log/run
5587 - update content in file /opt/gitlab/sv/alertmanager/log/run from none to 9a4626
5588 --- /opt/gitlab/sv/alertmanager/log/run 2023-07-20 03:23:07.000000000 +0000
5589 +++ /opt/gitlab/sv/alertmanager/log/.chef-run20230720-37-qlnt0a 2023-07-20 03:23:07.000000000 +0000
5590 @@ -1,5 +1,10 @@
5591 +#!/bin/sh
5592 +exec chpst -P \
5593 + -U root:root \
5594 + -u root:root \
5595 + svlogd -tt /var/log/gitlab/alertmanager[2023-07-20T03:23:07+00:00] INFO: template[/opt/gitlab/sv/alertmanager/log/run] owner changed to 0
5596[2023-07-20T03:23:07+00:00] INFO: template[/opt/gitlab/sv/alertmanager/log/run] group changed to 0
5597[2023-07-20T03:23:07+00:00] INFO: template[/opt/gitlab/sv/alertmanager/log/run] mode changed to 755
5598 - change mode from '' to '0755'
5599 - change owner from '' to 'root'
5600 - change group from '' to 'root'
5601 * directory[/opt/gitlab/sv/alertmanager/env] action create[2023-07-20T03:23:07+00:00] INFO: directory[/opt/gitlab/sv/alertmanager/env] created directory /opt/gitlab/sv/alertmanager/env
5602 - create new directory /opt/gitlab/sv/alertmanager/env[2023-07-20T03:23:07+00:00] INFO: directory[/opt/gitlab/sv/alertmanager/env] owner changed to 0
5603[2023-07-20T03:23:07+00:00] INFO: directory[/opt/gitlab/sv/alertmanager/env] group changed to 0
5604[2023-07-20T03:23:07+00:00] INFO: directory[/opt/gitlab/sv/alertmanager/env] mode changed to 755
5605 - change mode from '' to '0755'
5606 - change owner from '' to 'root'
5607 - change group from '' to 'root'
5608 * ruby_block[Delete unmanaged env files for alertmanager service] action run (skipped due to only_if)
5609 * template[/opt/gitlab/sv/alertmanager/check] action create (skipped due to only_if)
5610 * template[/opt/gitlab/sv/alertmanager/finish] action create (skipped due to only_if)
5611 * directory[/opt/gitlab/sv/alertmanager/control] action create[2023-07-20T03:23:07+00:00] INFO: directory[/opt/gitlab/sv/alertmanager/control] created directory /opt/gitlab/sv/alertmanager/control
5612 - create new directory /opt/gitlab/sv/alertmanager/control[2023-07-20T03:23:07+00:00] INFO: directory[/opt/gitlab/sv/alertmanager/control] owner changed to 0
5613[2023-07-20T03:23:07+00:00] INFO: directory[/opt/gitlab/sv/alertmanager/control] group changed to 0
5614[2023-07-20T03:23:07+00:00] INFO: directory[/opt/gitlab/sv/alertmanager/control] mode changed to 755
5615 - change mode from '' to '0755'
5616 - change owner from '' to 'root'
5617 - change group from '' to 'root'
5618 * link[/opt/gitlab/init/alertmanager] action create[2023-07-20T03:23:07+00:00] INFO: link[/opt/gitlab/init/alertmanager] created
5619 - create symlink at /opt/gitlab/init/alertmanager to /opt/gitlab/embedded/bin/sv
5620 * file[/opt/gitlab/sv/alertmanager/down] action nothing (skipped due to action :nothing)
5621[2023-07-20T03:23:07+00:00] INFO: template[/opt/gitlab/sv/alertmanager/run] sending run action to ruby_block[restart_service] (delayed)
5622 * ruby_block[restart_service] action run (skipped due to only_if)
5623[2023-07-20T03:23:07+00:00] INFO: template[/opt/gitlab/sv/alertmanager/log/config] sending create action to ruby_block[reload_log_service] (delayed)
5624 * ruby_block[reload_log_service] action create
5625 * ruby_block[restart_service] action nothing (skipped due to action :nothing)
5626 * ruby_block[restart_log_service] action nothing (skipped due to action :nothing)
5627 * ruby_block[reload_log_service] action nothing (skipped due to action :nothing)
5628 * directory[/opt/gitlab/sv/alertmanager] action create (up to date)
5629 * template[/opt/gitlab/sv/alertmanager/run] action create (up to date)
5630 * directory[/opt/gitlab/sv/alertmanager/log] action create (up to date)
5631 * directory[/opt/gitlab/sv/alertmanager/log/main] action create (up to date)
5632 * template[/opt/gitlab/sv/alertmanager/log/config] action create (up to date)
5633 * ruby_block[verify_chown_persisted_on_alertmanager] action nothing (skipped due to action :nothing)
5634 * link[/var/log/gitlab/alertmanager/config] action create (up to date)
5635 * template[/opt/gitlab/sv/alertmanager/log/run] action create (up to date)
5636 * directory[/opt/gitlab/sv/alertmanager/env] action create (up to date)
5637 * ruby_block[Delete unmanaged env files for alertmanager service] action run (skipped due to only_if)
5638 * template[/opt/gitlab/sv/alertmanager/check] action create (skipped due to only_if)
5639 * template[/opt/gitlab/sv/alertmanager/finish] action create (skipped due to only_if)
5640 * directory[/opt/gitlab/sv/alertmanager/control] action create (up to date)
5641 * link[/opt/gitlab/init/alertmanager] action create (up to date)
5642 * file[/opt/gitlab/sv/alertmanager/down] action nothing (skipped due to action :nothing)
5643 * directory[/opt/gitlab/service] action create (up to date)
5644 * link[/opt/gitlab/service/alertmanager] action create[2023-07-20T03:23:07+00:00] INFO: link[/opt/gitlab/service/alertmanager] created
5645 - create symlink at /opt/gitlab/service/alertmanager to /opt/gitlab/sv/alertmanager
5646 * ruby_block[wait for alertmanager service socket] action run[2023-07-20T03:23:11+00:00] INFO: ruby_block[wait for alertmanager service socket] called
5647 - execute the ruby block wait for alertmanager service socket
5648 * file[/var/log/gitlab/alertmanager/current] action touch (skipped due to only_if)
5649[2023-07-20T03:23:12+00:00] INFO: ruby_block[reload_log_service] called
5650 - execute the ruby block reload_log_service
5651[2023-07-20T03:23:12+00:00] INFO: template[/opt/gitlab/sv/alertmanager/log/run] sending run action to ruby_block[restart_log_service] (delayed)
5652 * ruby_block[restart_log_service] action run
5653 * ruby_block[restart_service] action nothing (skipped due to action :nothing)
5654 * ruby_block[restart_log_service] action nothing (skipped due to action :nothing)
5655 * ruby_block[reload_log_service] action nothing (skipped due to action :nothing)
5656 * directory[/opt/gitlab/sv/alertmanager] action create (up to date)
5657 * template[/opt/gitlab/sv/alertmanager/run] action create (up to date)
5658 * directory[/opt/gitlab/sv/alertmanager/log] action create (up to date)
5659 * directory[/opt/gitlab/sv/alertmanager/log/main] action create (up to date)
5660 * template[/opt/gitlab/sv/alertmanager/log/config] action create (up to date)
5661 * ruby_block[verify_chown_persisted_on_alertmanager] action nothing (skipped due to action :nothing)
5662 * link[/var/log/gitlab/alertmanager/config] action create (up to date)
5663 * template[/opt/gitlab/sv/alertmanager/log/run] action create (up to date)
5664 * directory[/opt/gitlab/sv/alertmanager/env] action create (up to date)
5665 * ruby_block[Delete unmanaged env files for alertmanager service] action run (skipped due to only_if)
5666 * template[/opt/gitlab/sv/alertmanager/check] action create (skipped due to only_if)
5667 * template[/opt/gitlab/sv/alertmanager/finish] action create (skipped due to only_if)
5668 * directory[/opt/gitlab/sv/alertmanager/control] action create (up to date)
5669 * link[/opt/gitlab/init/alertmanager] action create (up to date)
5670 * file[/opt/gitlab/sv/alertmanager/down] action nothing (skipped due to action :nothing)
5671 * directory[/opt/gitlab/service] action create (up to date)
5672 * link[/opt/gitlab/service/alertmanager] action create (up to date)
5673 * ruby_block[wait for alertmanager service socket] action run (skipped due to not_if)
5674 * file[/var/log/gitlab/alertmanager/current] action touch (skipped due to only_if)
5675[2023-07-20T03:23:12+00:00] INFO: ruby_block[restart_log_service] called
5676 - execute the ruby block restart_log_service
5677 * directory[/opt/gitlab/service] action create (up to date)
5678 * link[/opt/gitlab/service/alertmanager] action create (up to date)
5679 * ruby_block[wait for alertmanager service socket] action run (skipped due to not_if)
5680 * file[/var/log/gitlab/alertmanager/current] action touch (skipped due to only_if)
5681
5682 * execute[/opt/gitlab/bin/gitlab-ctl start alertmanager] action run
5683 [execute] ok: run: alertmanager: (pid 774) 3s
5684[2023-07-20T03:23:13+00:00] INFO: execute[/opt/gitlab/bin/gitlab-ctl start alertmanager] ran successfully
5685 - execute /opt/gitlab/bin/gitlab-ctl start alertmanager
5686Recipe: monitoring::postgres-exporter_disable
5687 * service[postgres-exporter] action nothing (skipped due to action :nothing)
5688 * runit_service[postgres-exporter] action disable
5689 * ruby_block[disable postgres-exporter] action run (skipped due to only_if)
5690 (up to date)
5691 * consul_service[postgres-exporter] action delete
5692 * file[/var/opt/gitlab/consul/config.d/postgres-exporter-service.json] action delete (up to date)
5693 (up to date)
5694Recipe: monitoring::grafana_disable
5695 * service[grafana] action nothing (skipped due to action :nothing)
5696 * runit_service[grafana] action disable
5697 * ruby_block[disable grafana] action run (skipped due to only_if)
5698 (up to date)
5699Recipe: gitlab::database_reindexing_disable
5700 * crond_job[database-reindexing] action delete
5701 * file[/var/opt/gitlab/crond/database-reindexing] action delete (up to date)
5702 (up to date)
5703[2023-07-20T03:23:13+00:00] INFO: templatesymlink[Create a gitlab.yml and create a symlink to Rails root] sending run action to execute[clear the gitlab-rails cache] (delayed)
5704Recipe: gitlab::gitlab-rails
5705 * execute[clear the gitlab-rails cache] action run (skipped due to not_if)
5706[2023-07-20T03:23:13+00:00] INFO: template[Create praefect config.toml] sending hup action to runit_service[praefect] (delayed)
5707Recipe: praefect::enable
5708 * runit_service[praefect] action hup[2023-07-20T03:23:13+00:00] INFO: runit_service[praefect] signalled (HUP)
5709[2023-07-20T03:23:13+00:00] INFO: runit_service[praefect] sent hup
5710 - send hup to runit_service[praefect]
5711[2023-07-20T03:23:13+00:00] INFO: version_file[Create version file for Gitlab KAS] sending restart action to runit_service[gitlab-kas] (delayed)
5712Recipe: gitlab-kas::enable
5713 * runit_service[gitlab-kas] action restart
5714
5715 ================================================================================
5716 Error executing action `restart` on resource 'runit_service[gitlab-kas]'
5717 ================================================================================
5718
5719 Mixlib::ShellOut::ShellCommandFailed
5720 ------------------------------------
5721 Expected process to exit with [0], but received '1'
5722 ---- Begin output of /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/gitlab-kas ----
5723 STDOUT: timeout: down: /opt/gitlab/service/gitlab-kas: 0s, normally up, want up
5724 STDERR:
5725 ---- End output of /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/gitlab-kas ----
5726 Ran /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/gitlab-kas returned 1
5727
5728 Cookbook Trace: (most recent call first)
5729 ----------------------------------------
5730 /opt/gitlab/embedded/cookbooks/cache/cookbooks/runit/libraries/helpers.rb:136:in `safe_sv_shellout!'
5731 /opt/gitlab/embedded/cookbooks/cache/cookbooks/runit/libraries/helpers.rb:164:in `restart_service'
5732 /opt/gitlab/embedded/cookbooks/cache/cookbooks/runit/libraries/provider_runit_service.rb:368:in `block in <class:RunitService>'
5733
5734 Resource Declaration:
5735 ---------------------
5736 suppressed sensitive resource output
5737
5738 Compiled Resource:
5739 ------------------
5740 suppressed sensitive resource output
5741
5742 System Info:
5743 ------------
5744 chef_version=17.10.0
5745 platform=ubuntu
5746 platform_version=22.04
5747 ruby=ruby 3.0.6p216 (2023-03-30 revision 23a532679b) [x86_64-linux]
5748 program_name=/opt/gitlab/embedded/bin/cinc-client
5749 executable=/opt/gitlab/embedded/bin/cinc-client
5750
5751[2023-07-20T03:23:44+00:00] INFO: env_dir[/opt/gitlab/etc/prometheus/env] sending restart action to runit_service[prometheus] (delayed)
5752Recipe: monitoring::prometheus
5753 * runit_service[prometheus] action restart (up to date)
5754[2023-07-20T03:23:44+00:00] INFO: file[Prometheus config] sending run action to execute[reload prometheus] (delayed)
5755 * execute[reload prometheus] action run[2023-07-20T03:23:45+00:00] INFO: execute[reload prometheus] ran successfully
5756 - execute /opt/gitlab/bin/gitlab-ctl hup prometheus
5757[2023-07-20T03:23:45+00:00] INFO: env_dir[/opt/gitlab/etc/alertmanager/env] sending restart action to runit_service[alertmanager] (delayed)
5758Recipe: monitoring::alertmanager
5759 * runit_service[alertmanager] action restart (up to date)
5760Running handlers:
5761[2023-07-20T03:23:45+00:00] ERROR: Running exception handlers
5762There was an error running gitlab-ctl reconfigure:
5763runit_service[gitlab-kas] (gitlab-kas::enable line 148) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
5764---- Begin output of /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/gitlab-kas ----
5765STDOUT: timeout: down: /opt/gitlab/service/gitlab-kas: 0s, normally up, want up
5766STDERR:
5767---- End output of /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/gitlab-kas ----
5768Ran /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/gitlab-kas returned 1
5769Running handlers complete
5770[2023-07-20T03:23:45+00:00] ERROR: Exception handlers complete
5771Infra Phase failed. 254 resources updated in 01 minutes 12 seconds
5772[2023-07-20T03:23:45+00:00] FATAL: Stacktrace dumped to /opt/gitlab/embedded/cookbooks/cache/cinc-stacktrace.out
5773[2023-07-20T03:23:45+00:00] FATAL: ---------------------------------------------------------------------------------------
5774[2023-07-20T03:23:45+00:00] FATAL: PLEASE PROVIDE THE CONTENTS OF THE stacktrace.out FILE (above) IF YOU FILE A BUG REPORT
5775[2023-07-20T03:23:45+00:00] FATAL: ---------------------------------------------------------------------------------------
5776[2023-07-20T03:23:45+00:00] FATAL: Mixlib::ShellOut::ShellCommandFailed: runit_service[gitlab-kas] (gitlab-kas::enable line 148) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
5777---- Begin output of /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/gitlab-kas ----
5778STDOUT: timeout: down: /opt/gitlab/service/gitlab-kas: 0s, normally up, want up
5779STDERR:
5780---- End output of /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/gitlab-kas ----
5781Ran /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/gitlab-kas returned 1
5782[Jul 20 2023 03:23:46 UTC (Gitlab QA)] WARN -- Retry instance_no_teardown due to Support::ShellCommand::StatusError -- attempt 1
5783[Jul 20 2023 03:23:46 UTC (Gitlab QA)] INFO -- Shell command: `docker ps -f name=praefect`
5784[Jul 20 2023 03:23:46 UTC (Gitlab QA)] INFO -- Shell command: `docker login --username "gitlab-ci-token" --password "*****" registry.gitlab.com`
5785[Jul 20 2023 03:23:46 UTC (Gitlab QA)] INFO -- Shell command: `docker pull -q registry.gitlab.com/gitlab-org/build/omnibus-gitlab-mirror/gitlab-ce:3913740266a59293daf9335e77aacbda3b1a42f1`
5786[Jul 20 2023 03:23:46 UTC (Gitlab QA)] INFO -- Shell command: `docker container list --all --format '{{.Names}}' --filter name=^praefect$`
5787[Jul 20 2023 03:23:46 UTC (Gitlab QA)] INFO -- Shell command: `docker rm -f praefect`
5788[Jul 20 2023 03:23:46 UTC (Gitlab QA)] INFO -- Shell command: `docker network list --format '{{.Name}}' --filter name=^test$`
5789[Jul 20 2023 03:23:46 UTC (Gitlab QA)] INFO -- Shell command: `docker run -d --name praefect --net test --hostname praefect.test -p 2305 --volume /builds/gitlab-org/gitlab/gitlab-qa-run-2023-07-20-03-13-21-3545a9c0/praefect/logs:/var/log/gitlab:z registry.gitlab.com/gitlab-org/build/omnibus-gitlab-mirror/gitlab-ce:3913740266a59293daf9335e77aacbda3b1a42f1`
5790[Jul 20 2023 03:23:47 UTC (Gitlab QA)] INFO -- Shell command: `docker exec praefect bash -c "echo \"postgresql['enable'] = false;
5791redis['enable'] = false;
5792nginx['enable'] = false;
5793puma['enable'] = false;
5794sidekiq['enable'] = false;
5795gitlab_workhorse['enable'] = false;
5796gitlab_rails['rake_cache_clear'] = false;
5797gitlab_rails['auto_migrate'] = false;
5798gitlab_exporter['enable'] = false;
5799gitaly['enable'] = false;
5800prometheus['enable'] = true;
5801praefect['enable'] = true;
5802praefect['configuration'] = {
5803 listen_addr: '0.0.0.0:2305',
5804 prometheus_listen_addr: '0.0.0.0:9652',
5805 auth: {
5806 token: 'PRAEFECT_EXTERNAL_TOKEN'
5807 },
5808 reconciliation: {
5809 scheduling_interval: '10s'
5810 },
5811 database: {
5812 host: 'postgres.test',
5813 port: 5432,
5814 user: 'postgres',
5815 password: 'SQL_PASSWORD',
5816 dbname: 'praefect_production',
5817 sslmode: 'disable'
5818 },
5819 virtual_storage: [
5820 {
5821 name: 'default',
5822 node: [
5823 {
5824 'storage': 'gitaly1',
5825 'address': 'tcp://gitaly1.test:8075',
5826 'token': 'PRAEFECT_INTERNAL_TOKEN'
5827 },
5828 {
5829 'storage': 'gitaly2',
5830 'address': 'tcp://gitaly2.test:8075',
5831 'token': 'PRAEFECT_INTERNAL_TOKEN'
5832 },
5833 {
5834 'storage': 'gitaly3',
5835 'address': 'tcp://gitaly3.test:8075',
5836 'token': 'PRAEFECT_INTERNAL_TOKEN'
5837 }
5838 ],
5839 }
5840 ]
5841}
5842\" > /etc/gitlab/gitlab.rb;"`
5843[Jul 20 2023 03:23:47 UTC (Gitlab QA)] INFO -- Shell command: `docker attach --sig-proxy=false praefect`
5844[Jul 20 2023 03:24:52 UTC (Gitlab QA)] INFO -- Shell command: `docker exec praefect bash -c "praefect -config /var/opt/gitlab/praefect/cluster_config.toml check || true"`
5845[Jul 20 2023 03:24:52 UTC (Gitlab QA)] INFO -- Shell command: `docker inspect -f '{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' praefect`
5846[Jul 20 2023 03:24:52 UTC (Gitlab QA)] INFO -- Gitaly Cluster Ready
5847[Jul 20 2023 03:24:52 UTC (Gitlab QA)] INFO -- Running Gitaly Cluster specs!
5848[Jul 20 2023 03:24:52 UTC (Gitlab QA)] INFO -- Shell command: `docker login --username "gitlab-ci-token" --password "*****" registry.gitlab.com`
5849[Jul 20 2023 03:24:52 UTC (Gitlab QA)] INFO -- Shell command: `docker pull -q registry.gitlab.com/gitlab-org/gitlab/gitlab-ce-qa:3913740266a59293daf9335e77aacbda3b1a42f1`
5850[Jul 20 2023 03:25:50 UTC (Gitlab QA)] INFO -- Running test suite `Test::Instance::All` for gitlab-ce
5851[Jul 20 2023 03:25:50 UTC (Gitlab QA)] INFO -- Shell command: `docker run -t --rm --net=test --env QA_AWS_S3_REGION="$QA_AWS_S3_REGION" --env QA_FACEBOOK_OAUTH_APP_ID="$QA_FACEBOOK_OAUTH_APP_ID" --env QA_GOOGLE_CDN_JSON_KEY="$QA_GOOGLE_CDN_JSON_KEY" --env QA_SUITE_STATUS_ENV_FILE="$QA_SUITE_STATUS_ENV_FILE" --env QA_THIRD_PARTY_DOCKER_REGISTRY="$QA_THIRD_PARTY_DOCKER_REGISTRY" --env QA_SAVE_TEST_METRICS="$QA_SAVE_TEST_METRICS" --env QA_GOOGLE_CDN_SIGNURL_KEY_NAME="$QA_GOOGLE_CDN_SIGNURL_KEY_NAME" --env QA_1P_PASSWORD="$QA_1P_PASSWORD" --env QA_REMOTE_GRID_USERNAME="$QA_REMOTE_GRID_USERNAME" --env QA_GOOGLE_CDN_SIGNURL_KEY="$QA_GOOGLE_CDN_SIGNURL_KEY" --env QA_TEST_CASE_PROJECT_TOKEN="$QA_TEST_CASE_PROJECT_TOKEN" --env QA_GITHUB_USERNAME="$QA_GITHUB_USERNAME" --env QA_SLOW_CONNECTION_THROUGHPUT_KBPS="$QA_SLOW_CONNECTION_THROUGHPUT_KBPS" --env QA_GITHUB_PASSWORD="$QA_GITHUB_PASSWORD" --env QA_ARTIFACTS_DIR="$QA_ARTIFACTS_DIR" --env QA_GITLAB_TLS_CERTIFICATE="$QA_GITLAB_TLS_CERTIFICATE" --env QA_RUN_TYPE="$QA_RUN_TYPE" --env QA_JIRA_ADMIN_PASSWORD="$QA_JIRA_ADMIN_PASSWORD" --env QA_GITHUB_OAUTH_APP_SECRET="$QA_GITHUB_OAUTH_APP_SECRET" --env QA_PATH="$QA_PATH" --env QA_JIRA_ADMIN_USERNAME="$QA_JIRA_ADMIN_USERNAME" --env QA_ADMIN_ACCESS_TOKEN="$QA_ADMIN_ACCESS_TOKEN" --env QA_COMMAND="$QA_COMMAND" --env QA_EE_LICENSE="$QA_EE_LICENSE" --env QA_1P_EMAIL="$QA_1P_EMAIL" --env QA_EE_ACTIVATION_CODE="$QA_EE_ACTIVATION_CODE" --env QA_GOOGLE_CLIENT_EMAIL="$QA_GOOGLE_CLIENT_EMAIL" --env QA_ALLURE_AWS_SECRET_ACCESS_KEY="$QA_ALLURE_AWS_SECRET_ACCESS_KEY" --env QA_GCS_BUCKET_NAME="$QA_GCS_BUCKET_NAME" --env QA_FACEBOOK_OAUTH_APP_SECRET="$QA_FACEBOOK_OAUTH_APP_SECRET" --env QA_GOOGLE_PROJECT="$QA_GOOGLE_PROJECT" --env QA_KNAPSACK_REPORT_GCS_CREDENTIALS="$QA_KNAPSACK_REPORT_GCS_CREDENTIALS" --env QA_AWS_S3_KEY_ID="$QA_AWS_S3_KEY_ID" --env QA_RUN_ALL_TESTS="$QA_RUN_ALL_TESTS" --env QA_GITLAB_CI_TOKEN="$QA_GITLAB_CI_TOKEN" --env QA_REMOTE_GRID_ACCESS_KEY="$QA_REMOTE_GRID_ACCESS_KEY" --env QA_FRAMEWORK_CHANGES="$QA_FRAMEWORK_CHANGES" --env QA_INTERCEPT_REQUESTS="$QA_INTERCEPT_REQUESTS" --env QA_INFLUXDB_URL="$QA_INFLUXDB_URL" --env QA_THIRD_PARTY_DOCKER_USER="$QA_THIRD_PARTY_DOCKER_USER" --env QA_THIRD_PARTY_DOCKER_PASSWORD="$QA_THIRD_PARTY_DOCKER_PASSWORD" --env QA_1P_SECRET="$QA_1P_SECRET" --env QA_AWS_S3_ACCESS_KEY="$QA_AWS_S3_ACCESS_KEY" --env QA_LOG_LEVEL="$QA_LOG_LEVEL" --env QA_GITHUB_OAUTH_APP_ID="$QA_GITHUB_OAUTH_APP_ID" --env QA_EXPORT_TEST_METRICS="$QA_EXPORT_TEST_METRICS" --env QA_GOOGLE_CDN_LB="$QA_GOOGLE_CDN_LB" --env QA_1P_GITHUB_UUID="$QA_1P_GITHUB_UUID" --env QA_FACEBOOK_PASSWORD="$QA_FACEBOOK_PASSWORD" --env QA_GCS_CDN_BUCKET_NAME="$QA_GCS_CDN_BUCKET_NAME" --env QA_SCENARIO="$QA_SCENARIO" --env QA_GOOGLE_JSON_KEY="$QA_GOOGLE_JSON_KEY" --env QA_TRIGGER_AUTO_QUARANTINE="$QA_TRIGGER_AUTO_QUARANTINE" --env QA_SUITES="$QA_SUITES" --env QA_RUN_ALL_E2E_LABEL="$QA_RUN_ALL_E2E_LABEL" --env QA_ALLURE_AWS_ACCESS_KEY_ID="$QA_ALLURE_AWS_ACCESS_KEY_ID" --env QA_AWS_S3_BUCKET_NAME="$QA_AWS_S3_BUCKET_NAME" --env QA_GENERATE_ALLURE_REPORT="$QA_GENERATE_ALLURE_REPORT" --env QA_CAN_TEST_PRAEFECT="$QA_CAN_TEST_PRAEFECT" --env QA_RUN_TESTS_ON_GDK="$QA_RUN_TESTS_ON_GDK" --env QA_INFLUXDB_TOKEN="$QA_INFLUXDB_TOKEN" --env QA_TEST_LICENSE_MODE="$QA_TEST_LICENSE_MODE" --env QA_TEST_SESSION_TOKEN="$QA_TEST_SESSION_TOKEN" --env QA_SLOW_CONNECTION_LATENCY_MS="$QA_SLOW_CONNECTION_LATENCY_MS" --env QA_RESULTS_ISSUE_PROJECT_TOKEN="$QA_RESULTS_ISSUE_PROJECT_TOKEN" --env QA_GITHUB_ACCESS_TOKEN="$QA_GITHUB_ACCESS_TOKEN" --env QA_FACEBOOK_USERNAME="$QA_FACEBOOK_USERNAME" --env QA_THIRD_PARTY_DOCKER_REPOSITORY="$QA_THIRD_PARTY_DOCKER_REPOSITORY" --env QA_SELENOID_BROWSER_VERSION="111.0" --env QA_VIDEO_RECORDER_IMAGE="presidenten/selenoid-manual-video-recorder" --env QA_VIDEO_RECORDER_VERSION="latest" --env QA_SELENOID_BROWSER_IMAGE="selenoid/chrome" --env GITLAB_QA_ADMIN_ACCESS_TOKEN="$GITLAB_QA_ADMIN_ACCESS_TOKEN" --env GITLAB_QA_PASSWORD_1="$GITLAB_QA_PASSWORD_1" --env GITLAB_QA_PASSWORD_2="$GITLAB_QA_PASSWORD_2" --env CI="$CI" --env CI_JOB_NAME="$CI_JOB_NAME" --env CI_JOB_NAME_SLUG="$CI_JOB_NAME_SLUG" --env CI_JOB_URL="$CI_JOB_URL" --env CI_JOB_TOKEN="$CI_JOB_TOKEN" --env CI_RUNNER_ID="$CI_RUNNER_ID" --env CI_SERVER_HOST="$CI_SERVER_HOST" --env CI_NODE_TOTAL="$CI_NODE_TOTAL" --env CI_PROJECT_NAME="$CI_PROJECT_NAME" --env CI_PROJECT_PATH="$CI_PROJECT_PATH" --env CI_PIPELINE_SOURCE="$CI_PIPELINE_SOURCE" --env CI_PIPELINE_URL="$CI_PIPELINE_URL" --env CI_PIPELINE_CREATED_AT="$CI_PIPELINE_CREATED_AT" --env GITLAB_CI="$GITLAB_CI" --env GITLAB_QA_USER_AGENT="$GITLAB_QA_USER_AGENT" --env GITLAB_INITIAL_ROOT_PASSWORD="5iveL!fe" --env COLORIZED_LOGS="$COLORIZED_LOGS" --env RELEASE="$RELEASE" --env RELEASE_REGISTRY_URL="registry.gitlab.com" --env RELEASE_REGISTRY_USERNAME="gitlab-ci-token" --env RELEASE_REGISTRY_PASSWORD="[MASKED]" --env SELENOID_DIRECTORY="/builds/gitlab-org/gitlab/gitlab-qa-run-2023-07-20-03-13-21-3545a9c0/selenoid" --env SCHEDULE_TYPE="$SCHEDULE_TYPE" --volume /builds/gitlab-org/gitlab/gitlab-qa-run-2023-07-20-03-13-21-3545a9c0/gitlab-ce-qa-e1e856a1:/home/gitlab/qa/tmp:z --volume /var/run/docker.sock:/var/run/docker.sock:z --name gitlab-ce-qa-e1e856a1 registry.gitlab.com/gitlab-org/gitlab/gitlab-ce-qa:3913740266a59293daf9335e77aacbda3b1a42f1 Test::Instance::All http://gitlab-gitaly-cluster.test -- --format QA::Support::JsonFormatter --out tmp/rspec-4694998482.json --format RspecJunitFormatter --out tmp/rspec-4694998482.xml --format html --out tmp/rspec-gitaly-cluster.htm --color --format documentation --tag gitaly_cluster`
58522023-07-20 03:25:56 / CONF ::
5853 ==> Base URL: http://gitlab-gitaly-cluster.test
5854 ==> Browser: #<Selenium::WebDriver::Chrome::Driver:0x00007f8f8a441fa8>
5855 ==> Libraries: Chemlab::Vendor
5856[Jul 20 2023 03:25:56 UTC (QA Tests)] INFO -- Caching token for username: root, last six chars of token:sixwPA
5857[Jul 20 2023 03:25:56 UTC (QA Tests)] INFO -- Browser: chrome
5858[Jul 20 2023 03:25:56 UTC (QA Tests)] INFO -- Performing sanity check for environment!
5859Run options:
5860 include {:gitaly_cluster=>true}
5861 exclude {:geo=>true, :requires_praefect=>true}
5862Randomized with seed 48179
5863Systems
5864 with Gitaly automatic failover and recovery
5865[Jul 20 2023 03:25:59 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} postgres`
5866[Jul 20 2023 03:25:59 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT 1 as healthy_database\""
5867`
5868[Jul 20 2023 03:25:59 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly1`
5869[Jul 20 2023 03:25:59 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly2`
5870[Jul 20 2023 03:25:59 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly3`
5871[Jul 20 2023 03:26:00 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} praefect`
5872[Jul 20 2023 03:26:00 UTC (QA Tests)] INFO -- Waiting for health check on praefect
5873[Jul 20 2023 03:26:00 UTC (QA Tests)] INFO -- Executing: `docker exec praefect gitlab-ctl status praefect`
5874[Jul 20 2023 03:26:00 UTC (QA Tests)] INFO -- Waiting for health check on gitaly1
5875[Jul 20 2023 03:26:00 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly1 gitlab-ctl status gitaly`
5876[Jul 20 2023 03:26:01 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
5877`
5878[Jul 20 2023 03:26:01 UTC (QA Tests)] INFO -- Waiting for health check on gitaly2
5879[Jul 20 2023 03:26:01 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly2 gitlab-ctl status gitaly`
5880[Jul 20 2023 03:26:02 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
5881`
5882[Jul 20 2023 03:26:02 UTC (QA Tests)] INFO -- Waiting for health check on gitaly3
5883[Jul 20 2023 03:26:02 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly3 gitlab-ctl status gitaly`
5884[Jul 20 2023 03:26:03 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""
5885`
5886[Jul 20 2023 03:26:03 UTC (QA Tests)] INFO -- Retrieved cached token for username: root, last six chars of token:sixwPA
5887[Jul 20 2023 03:26:03 UTC (QA Tests)] INFO -- ==> Retrieved a QA::Resource::PersonalAccessToken with name 'api-pat-root-zwmoxalt' via cache in 0.0 seconds
5888[Jul 20 2023 03:26:04 UTC (QA Tests)] INFO -- ==> Built a QA::Resource::Sandbox with full_path 'gitlab-qa-sandbox-group-5' via api in 0.97 seconds
5889[Jul 20 2023 03:26:05 UTC (QA Tests)] INFO -- ==> Built a QA::Resource::Group with full_path 'gitlab-qa-sandbox-group-5/qa-test-2023-07-20-03-26-03-85db8ef7ea358904' via api in 1.9 seconds
5890[Jul 20 2023 03:26:07 UTC (QA Tests)] INFO -- ==> Built a QA::Resource::Project with full_path 'gitlab-qa-sandbox-group-5/qa-test-2023-07-20-03-26-03-85db8ef7ea358904/gitaly_cluster-3f7ec71e7c0633c8' via api in 3.98 seconds
5891[Jul 20 2023 03:26:07 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue where state = 'ready' or state = 'in_progress';\""
5892`
5893[Jul 20 2023 03:26:07 UTC (QA Tests)] INFO -- Executing: `docker exec gitlab-gitaly-cluster bash -c 'gitlab-rake "gitlab:praefect:replicas[1]"'`
5894[Jul 20 2023 03:26:38 UTC (QA Tests)] INFO -- Starting test: Systems with Gitaly automatic failover and recovery automatically fails over
5895[Jul 20 2023 03:26:38 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly2`
5896[Jul 20 2023 03:26:38 UTC (QA Tests)] INFO -- Executing: `docker pause gitaly2`
5897[Jul 20 2023 03:26:39 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
5898`
5899[Jul 20 2023 03:26:40 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
5900`
5901[Jul 20 2023 03:26:41 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
5902`
5903[Jul 20 2023 03:26:42 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
5904`
5905[Jul 20 2023 03:26:43 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
5906`
5907[Jul 20 2023 03:26:44 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
5908`
5909[Jul 20 2023 03:26:45 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
5910`
5911[Jul 20 2023 03:26:47 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
5912`
5913[Jul 20 2023 03:26:48 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
5914`
5915[Jul 20 2023 03:26:49 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
5916`
5917[Jul 20 2023 03:26:49 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly3`
5918[Jul 20 2023 03:26:49 UTC (QA Tests)] INFO -- Executing: `docker pause gitaly3`
5919[Jul 20 2023 03:26:49 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""
5920`
5921[Jul 20 2023 03:26:50 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""
5922`
5923[Jul 20 2023 03:26:52 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""
5924`
5925[Jul 20 2023 03:26:53 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""
5926`
5927[Jul 20 2023 03:26:54 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""
5928`
5929[Jul 20 2023 03:26:55 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""
5930`
5931[Jul 20 2023 03:26:56 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""
5932`
5933[Jul 20 2023 03:26:57 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""
5934`
5935[Jul 20 2023 03:26:58 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""
5936`
5937[Jul 20 2023 03:27:01 UTC (QA Tests)] INFO -- QA::Resource::Project - wait_for_push with commit message "first_added_commit_message to primary gitaly node"
5938[Jul 20 2023 03:27:01 UTC (QA Tests)] INFO -- ==> Built a QA::Resource::Repository::ProjectPush via browser_ui in 2.48 seconds
5939[Jul 20 2023 03:27:01 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} postgres`
5940[Jul 20 2023 03:27:01 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT 1 as healthy_database\""
5941`
5942[Jul 20 2023 03:27:01 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly1`
5943[Jul 20 2023 03:27:01 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly2`
5944[Jul 20 2023 03:27:01 UTC (QA Tests)] INFO -- Executing: `docker unpause gitaly2`
5945[Jul 20 2023 03:27:02 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Running}} gitaly2`
5946[Jul 20 2023 03:27:02 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly3`
5947[Jul 20 2023 03:27:02 UTC (QA Tests)] INFO -- Executing: `docker unpause gitaly3`
5948[Jul 20 2023 03:27:02 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Running}} gitaly3`
5949[Jul 20 2023 03:27:02 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} praefect`
5950[Jul 20 2023 03:27:02 UTC (QA Tests)] INFO -- Waiting for health check on praefect
5951[Jul 20 2023 03:27:02 UTC (QA Tests)] INFO -- Executing: `docker exec praefect gitlab-ctl status praefect`
5952[Jul 20 2023 03:27:03 UTC (QA Tests)] INFO -- Waiting for health check on gitaly1
5953[Jul 20 2023 03:27:03 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly1 gitlab-ctl status gitaly`
5954[Jul 20 2023 03:27:04 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
5955`
5956[Jul 20 2023 03:27:04 UTC (QA Tests)] INFO -- Waiting for health check on gitaly2
5957[Jul 20 2023 03:27:04 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly2 gitlab-ctl status gitaly`
5958[Jul 20 2023 03:27:05 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
5959`
5960[Jul 20 2023 03:27:05 UTC (QA Tests)] INFO -- Waiting for health check on gitaly3
5961[Jul 20 2023 03:27:05 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly3 gitlab-ctl status gitaly`
5962[Jul 20 2023 03:27:06 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""
5963`
5964[Jul 20 2023 03:27:06 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue where state = 'ready' or state = 'in_progress';\""
5965`
5966[Jul 20 2023 03:27:06 UTC (QA Tests)] INFO -- Executing: `docker exec gitlab-gitaly-cluster bash -c 'gitlab-rake "gitlab:praefect:replicas[1]"'`
5967[Jul 20 2023 03:27:36 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly1`
5968[Jul 20 2023 03:27:36 UTC (QA Tests)] INFO -- Executing: `docker pause gitaly1`
5969[Jul 20 2023 03:27:36 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
5970`
5971[Jul 20 2023 03:27:37 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
5972`
5973[Jul 20 2023 03:27:38 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
5974`
5975[Jul 20 2023 03:27:39 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
5976`
5977[Jul 20 2023 03:27:40 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
5978`
5979[Jul 20 2023 03:27:42 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
5980`
5981[Jul 20 2023 03:27:43 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
5982`
5983[Jul 20 2023 03:27:44 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
5984`
5985[Jul 20 2023 03:27:45 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
5986`
5987[Jul 20 2023 03:27:45 UTC (QA Tests)] INFO -- Waiting for health check failure on gitaly1
5988[Jul 20 2023 03:27:45 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
5989`
5990[Jul 20 2023 03:27:47 UTC (QA Tests)] INFO -- ==> Built a QA::Resource::Repository::Commit via api in 1.32 seconds
5991 automatically fails over
5992 when recovering from dataloss after failover
5993[Jul 20 2023 03:27:47 UTC (QA Tests)] INFO -- Starting test: Systems with Gitaly automatic failover and recovery when recovering from dataloss after failover automatically reconciles
5994[Jul 20 2023 03:27:47 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly1`
5995[Jul 20 2023 03:27:47 UTC (QA Tests)] INFO -- Executing: `docker unpause gitaly1`
5996[Jul 20 2023 03:27:47 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Running}} gitaly1`
5997[Jul 20 2023 03:27:47 UTC (QA Tests)] INFO -- Waiting for health check on gitaly1
5998[Jul 20 2023 03:27:47 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly1 gitlab-ctl status gitaly`
5999[Jul 20 2023 03:27:48 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
6000`
6001[Jul 20 2023 03:27:48 UTC (QA Tests)] INFO -- Executing: `docker exec gitlab-gitaly-cluster bash -c 'gitlab-rake "gitlab:praefect:replicas[1]"'`
6002[Jul 20 2023 03:28:18 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} postgres`
6003[Jul 20 2023 03:28:19 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT 1 as healthy_database\""
6004`
6005[Jul 20 2023 03:28:19 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly1`
6006[Jul 20 2023 03:28:19 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly2`
6007[Jul 20 2023 03:28:19 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly3`
6008[Jul 20 2023 03:28:19 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} praefect`
6009[Jul 20 2023 03:28:19 UTC (QA Tests)] INFO -- Waiting for health check on praefect
6010[Jul 20 2023 03:28:19 UTC (QA Tests)] INFO -- Executing: `docker exec praefect gitlab-ctl status praefect`
6011[Jul 20 2023 03:28:20 UTC (QA Tests)] INFO -- Waiting for health check on gitaly1
6012[Jul 20 2023 03:28:20 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly1 gitlab-ctl status gitaly`
6013[Jul 20 2023 03:28:21 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
6014`
6015[Jul 20 2023 03:28:21 UTC (QA Tests)] INFO -- Waiting for health check on gitaly2
6016[Jul 20 2023 03:28:21 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly2 gitlab-ctl status gitaly`
6017[Jul 20 2023 03:28:22 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
6018`
6019[Jul 20 2023 03:28:22 UTC (QA Tests)] INFO -- Waiting for health check on gitaly3
6020[Jul 20 2023 03:28:22 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly3 gitlab-ctl status gitaly`
6021[Jul 20 2023 03:28:23 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""
6022`
6023 automatically reconciles
6024Systems
6025 Gitaly backend node recovery
6026[Jul 20 2023 03:28:23 UTC (QA Tests)] INFO -- Starting test: Systems Gitaly backend node recovery recovers from dataloss
6027[Jul 20 2023 03:28:23 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} postgres`
6028[Jul 20 2023 03:28:23 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT 1 as healthy_database\""
6029`
6030[Jul 20 2023 03:28:23 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly1`
6031[Jul 20 2023 03:28:23 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly2`
6032[Jul 20 2023 03:28:23 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly3`
6033[Jul 20 2023 03:28:23 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} praefect`
6034[Jul 20 2023 03:28:23 UTC (QA Tests)] INFO -- Waiting for health check on praefect
6035[Jul 20 2023 03:28:23 UTC (QA Tests)] INFO -- Executing: `docker exec praefect gitlab-ctl status praefect`
6036[Jul 20 2023 03:28:24 UTC (QA Tests)] INFO -- Waiting for health check on gitaly1
6037[Jul 20 2023 03:28:24 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly1 gitlab-ctl status gitaly`
6038[Jul 20 2023 03:28:25 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
6039`
6040[Jul 20 2023 03:28:25 UTC (QA Tests)] INFO -- Waiting for health check on gitaly2
6041[Jul 20 2023 03:28:25 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly2 gitlab-ctl status gitaly`
6042[Jul 20 2023 03:28:26 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
6043`
6044[Jul 20 2023 03:28:26 UTC (QA Tests)] INFO -- Waiting for health check on gitaly3
6045[Jul 20 2023 03:28:26 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly3 gitlab-ctl status gitaly`
6046[Jul 20 2023 03:28:27 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""
6047`
6048[Jul 20 2023 03:28:27 UTC (QA Tests)] INFO -- ==> Retrieved a QA::Resource::Sandbox with full_path 'gitlab-qa-sandbox-group-5' via api in 0.11 seconds
6049[Jul 20 2023 03:28:27 UTC (QA Tests)] INFO -- ==> Retrieved a QA::Resource::Group with full_path 'gitlab-qa-sandbox-group-5/qa-test-2023-07-20-03-26-03-85db8ef7ea358904' via api in 0.21 seconds
6050[Jul 20 2023 03:28:30 UTC (QA Tests)] INFO -- ==> Built a QA::Resource::Project with full_path 'gitlab-qa-sandbox-group-5/qa-test-2023-07-20-03-26-03-85db8ef7ea358904/gitaly_cluster-eba6c4f1aaa88b91' via api in 2.36 seconds
6051[Jul 20 2023 03:28:30 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue where state = 'ready' or state = 'in_progress';\""
6052`
6053[Jul 20 2023 03:28:30 UTC (QA Tests)] INFO -- Executing: `docker exec gitlab-gitaly-cluster bash -c 'gitlab-rake "gitlab:praefect:replicas[2]"'`
6054[Jul 20 2023 03:28:59 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly1`
6055[Jul 20 2023 03:28:59 UTC (QA Tests)] INFO -- Executing: `docker pause gitaly1`
6056[Jul 20 2023 03:28:59 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
6057`
6058[Jul 20 2023 03:29:01 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
6059`
6060[Jul 20 2023 03:29:02 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
6061`
6062[Jul 20 2023 03:29:03 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
6063`
6064[Jul 20 2023 03:29:04 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
6065`
6066[Jul 20 2023 03:29:05 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
6067`
6068[Jul 20 2023 03:29:06 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
6069`
6070[Jul 20 2023 03:29:08 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
6071`
6072[Jul 20 2023 03:29:09 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
6073`
6074[Jul 20 2023 03:29:10 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
6075`
6076[Jul 20 2023 03:29:10 UTC (QA Tests)] INFO -- Waiting for health check failure on gitaly1
6077[Jul 20 2023 03:29:10 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
6078`
6079[Jul 20 2023 03:29:12 UTC (QA Tests)] INFO -- QA::Resource::Project - wait_for_push with commit message "pushed after failover"
6080[Jul 20 2023 03:29:12 UTC (QA Tests)] INFO -- ==> Built a QA::Resource::Repository::ProjectPush via browser_ui in 2.16 seconds
6081[Jul 20 2023 03:29:12 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select job from replication_queue
6082where state = 'ready'
6083 and job ->> 'change' = 'update'
6084 and job ->> 'target_node_storage' = 'gitaly1';
6085\""
6086`
6087[Jul 20 2023 03:29:12 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly1`
6088[Jul 20 2023 03:29:13 UTC (QA Tests)] INFO -- Executing: `docker unpause gitaly1`
6089[Jul 20 2023 03:29:13 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Running}} gitaly1`
6090[Jul 20 2023 03:29:13 UTC (QA Tests)] INFO -- Waiting for health check on gitaly1
6091[Jul 20 2023 03:29:13 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly1 gitlab-ctl status gitaly`
6092[Jul 20 2023 03:29:14 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
6093`
6094[Jul 20 2023 03:29:14 UTC (QA Tests)] INFO -- Waiting for health check on gitaly2
6095[Jul 20 2023 03:29:14 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly2 gitlab-ctl status gitaly`
6096[Jul 20 2023 03:29:15 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
6097`
6098[Jul 20 2023 03:29:15 UTC (QA Tests)] INFO -- Waiting for health check on gitaly3
6099[Jul 20 2023 03:29:15 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly3 gitlab-ctl status gitaly`
6100[Jul 20 2023 03:29:15 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""
6101`
6102[Jul 20 2023 03:29:16 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue where state = 'ready' or state = 'in_progress';\""
6103`
6104[Jul 20 2023 03:29:16 UTC (QA Tests)] INFO -- Executing: `docker exec gitlab-gitaly-cluster bash -c 'gitlab-rake "gitlab:praefect:replicas[2]"'`
6105[Jul 20 2023 03:29:45 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly2`
6106[Jul 20 2023 03:29:45 UTC (QA Tests)] INFO -- Executing: `docker pause gitaly2`
6107[Jul 20 2023 03:29:45 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
6108`
6109[Jul 20 2023 03:29:46 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
6110`
6111[Jul 20 2023 03:29:48 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
6112`
6113[Jul 20 2023 03:29:49 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
6114`
6115[Jul 20 2023 03:29:50 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
6116`
6117[Jul 20 2023 03:29:51 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
6118`
6119[Jul 20 2023 03:29:52 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
6120`
6121[Jul 20 2023 03:29:53 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
6122`
6123[Jul 20 2023 03:29:55 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
6124`
6125[Jul 20 2023 03:29:56 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
6126`
6127[Jul 20 2023 03:29:56 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly3`
6128[Jul 20 2023 03:29:56 UTC (QA Tests)] INFO -- Executing: `docker pause gitaly3`
6129[Jul 20 2023 03:29:56 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""
6130`
6131[Jul 20 2023 03:29:57 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""
6132`
6133[Jul 20 2023 03:29:58 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""
6134`
6135[Jul 20 2023 03:29:59 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""
6136`
6137[Jul 20 2023 03:30:01 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""
6138`
6139[Jul 20 2023 03:30:02 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""
6140`
6141[Jul 20 2023 03:30:03 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""
6142`
6143[Jul 20 2023 03:30:04 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""
6144`
6145[Jul 20 2023 03:30:05 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""
6146`
6147[Jul 20 2023 03:30:06 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""
6148`
6149[Jul 20 2023 03:30:07 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""
6150`
6151[Jul 20 2023 03:30:08 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} postgres`
6152[Jul 20 2023 03:30:08 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT 1 as healthy_database\""
6153`
6154[Jul 20 2023 03:30:08 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly1`
6155[Jul 20 2023 03:30:08 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly2`
6156[Jul 20 2023 03:30:08 UTC (QA Tests)] INFO -- Executing: `docker unpause gitaly2`
6157[Jul 20 2023 03:30:08 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Running}} gitaly2`
6158[Jul 20 2023 03:30:08 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly3`
6159[Jul 20 2023 03:30:08 UTC (QA Tests)] INFO -- Executing: `docker unpause gitaly3`
6160[Jul 20 2023 03:30:08 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Running}} gitaly3`
6161[Jul 20 2023 03:30:09 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} praefect`
6162[Jul 20 2023 03:30:09 UTC (QA Tests)] INFO -- Waiting for health check on praefect
6163[Jul 20 2023 03:30:09 UTC (QA Tests)] INFO -- Executing: `docker exec praefect gitlab-ctl status praefect`
6164[Jul 20 2023 03:30:09 UTC (QA Tests)] INFO -- Waiting for health check on gitaly1
6165[Jul 20 2023 03:30:09 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly1 gitlab-ctl status gitaly`
6166[Jul 20 2023 03:30:10 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
6167`
6168[Jul 20 2023 03:30:10 UTC (QA Tests)] INFO -- Waiting for health check on gitaly2
6169[Jul 20 2023 03:30:10 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly2 gitlab-ctl status gitaly`
6170[Jul 20 2023 03:30:11 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
6171`
6172[Jul 20 2023 03:30:11 UTC (QA Tests)] INFO -- Waiting for health check on gitaly3
6173[Jul 20 2023 03:30:11 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly3 gitlab-ctl status gitaly`
6174[Jul 20 2023 03:30:12 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""
6175`
6176 recovers from dataloss
6177Systems
6178 Gitaly Cluster replication queue
6179[Jul 20 2023 03:30:12 UTC (QA Tests)] INFO -- Starting test: Systems Gitaly Cluster replication queue allows replication of different repository after interruption
6180[Jul 20 2023 03:30:12 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} postgres`
6181[Jul 20 2023 03:30:12 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT 1 as healthy_database\""
6182`
6183[Jul 20 2023 03:30:13 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly1`
6184[Jul 20 2023 03:30:13 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly2`
6185[Jul 20 2023 03:30:13 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly3`
6186[Jul 20 2023 03:30:13 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} praefect`
6187[Jul 20 2023 03:30:13 UTC (QA Tests)] INFO -- Waiting for health check on praefect
6188[Jul 20 2023 03:30:13 UTC (QA Tests)] INFO -- Executing: `docker exec praefect gitlab-ctl status praefect`
6189[Jul 20 2023 03:30:14 UTC (QA Tests)] INFO -- Waiting for health check on gitaly1
6190[Jul 20 2023 03:30:14 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly1 gitlab-ctl status gitaly`
6191[Jul 20 2023 03:30:15 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
6192`
6193[Jul 20 2023 03:30:15 UTC (QA Tests)] INFO -- Waiting for health check on gitaly2
6194[Jul 20 2023 03:30:15 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly2 gitlab-ctl status gitaly`
6195[Jul 20 2023 03:30:15 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
6196`
6197[Jul 20 2023 03:30:16 UTC (QA Tests)] INFO -- Waiting for health check on gitaly3
6198[Jul 20 2023 03:30:16 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly3 gitlab-ctl status gitaly`
6199[Jul 20 2023 03:30:16 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""
6200`
6201[Jul 20 2023 03:30:17 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly2`
6202[Jul 20 2023 03:30:17 UTC (QA Tests)] INFO -- Executing: `docker pause gitaly2`
6203[Jul 20 2023 03:30:17 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
6204`
6205[Jul 20 2023 03:30:18 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
6206`
6207[Jul 20 2023 03:30:19 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
6208`
6209[Jul 20 2023 03:30:20 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
6210`
6211[Jul 20 2023 03:30:21 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
6212`
6213[Jul 20 2023 03:30:22 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
6214`
6215[Jul 20 2023 03:30:24 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
6216`
6217[Jul 20 2023 03:30:25 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
6218`
6219[Jul 20 2023 03:30:26 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
6220`
6221[Jul 20 2023 03:30:27 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
6222`
6223[Jul 20 2023 03:30:27 UTC (QA Tests)] INFO -- ==> Retrieved a QA::Resource::Sandbox with full_path 'gitlab-qa-sandbox-group-5' via api in 0.07 seconds
6224[Jul 20 2023 03:30:27 UTC (QA Tests)] INFO -- ==> Retrieved a QA::Resource::Group with full_path 'gitlab-qa-sandbox-group-5/qa-test-2023-07-20-03-26-03-85db8ef7ea358904' via api in 0.13 seconds
6225[Jul 20 2023 03:30:29 UTC (QA Tests)] INFO -- ==> Built a QA::Resource::Project with full_path 'gitlab-qa-sandbox-group-5/qa-test-2023-07-20-03-26-03-85db8ef7ea358904/gitaly_cluster-413c858803bcc156' via api in 1.54 seconds
6226[Jul 20 2023 03:30:50 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly2`
6227[Jul 20 2023 03:30:50 UTC (QA Tests)] INFO -- Executing: `docker unpause gitaly2`
6228[Jul 20 2023 03:30:50 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Running}} gitaly2`
6229[Jul 20 2023 03:30:50 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue_lock where acquired = 't';\""
6230`
6231[Jul 20 2023 03:30:51 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue_lock where acquired = 't';\""
6232`
6233[Jul 20 2023 03:30:51 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue_lock where acquired = 't';\""
6234`
6235[Jul 20 2023 03:30:51 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue_lock where acquired = 't';\""
6236`
6237[Jul 20 2023 03:30:51 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue_lock where acquired = 't';\""
6238`
6239[Jul 20 2023 03:30:51 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue_lock where acquired = 't';\""
6240`
6241[Jul 20 2023 03:30:51 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue_lock where acquired = 't';\""
6242`
6243[Jul 20 2023 03:30:52 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue_lock where acquired = 't';\""
6244`
6245[Jul 20 2023 03:30:52 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue_lock where acquired = 't';\""
6246`
6247[Jul 20 2023 03:30:52 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue_lock where acquired = 't';\""
6248`
6249[Jul 20 2023 03:30:52 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue_lock where acquired = 't';\""
6250`
6251[Jul 20 2023 03:30:52 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue_lock where acquired = 't';\""
6252`
6253[Jul 20 2023 03:30:52 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue_lock where acquired = 't';\""
6254`
6255[Jul 20 2023 03:30:52 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue_lock where acquired = 't';\""
6256`
6257[Jul 20 2023 03:30:53 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue_lock where acquired = 't';\""
6258`
6259[Jul 20 2023 03:30:53 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue_lock where acquired = 't';\""
6260`
6261[Jul 20 2023 03:30:53 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue_lock where acquired = 't';\""
6262`
6263[Jul 20 2023 03:30:53 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue_lock where acquired = 't';\""
6264`
6265[Jul 20 2023 03:30:53 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue_lock where acquired = 't';\""
6266`
6267[Jul 20 2023 03:30:53 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue_lock where acquired = 't';\""
6268`
6269[Jul 20 2023 03:30:53 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue_lock where acquired = 't';\""
6270`
6271[Jul 20 2023 03:30:54 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue_lock where acquired = 't';\""
6272`
6273[Jul 20 2023 03:30:54 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue_lock where acquired = 't';\""
6274`
6275[Jul 20 2023 03:30:54 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue_lock where acquired = 't';\""
6276`
6277[Jul 20 2023 03:30:54 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue_lock where acquired = 't';\""
6278`
6279[Jul 20 2023 03:30:54 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue_lock where acquired = 't';\""
6280`
6281[Jul 20 2023 03:30:54 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue_lock where acquired = 't';\""
6282`
6283[Jul 20 2023 03:30:54 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue_lock where acquired = 't';\""
6284`
6285[Jul 20 2023 03:30:55 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue_lock where acquired = 't';\""
6286`
6287[Jul 20 2023 03:30:55 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue_lock where acquired = 't';\""
6288`
6289[Jul 20 2023 03:30:55 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue_lock where acquired = 't';\""
6290`
6291[Jul 20 2023 03:30:55 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue_lock where acquired = 't';\""
6292`
6293[Jul 20 2023 03:30:55 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue_lock where acquired = 't';\""
6294`
6295[Jul 20 2023 03:30:55 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue_lock where acquired = 't';\""
6296`
6297[Jul 20 2023 03:30:56 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} praefect`
6298[Jul 20 2023 03:30:56 UTC (QA Tests)] INFO -- Executing: `docker pause praefect`
6299[Jul 20 2023 03:30:56 UTC (QA Tests)] INFO -- Setting jobs in replication queue to `in_progress` and acquiring locks
6300[Jul 20 2023 03:30:56 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"update replication_queue set state = 'in_progress';
6301insert into replication_queue_job_lock (job_id, lock_id, triggered_at)
6302 select id, rq.lock_id, created_at from replication_queue rq
6303 left join replication_queue_job_lock rqjl on rq.id = rqjl.job_id
6304 where state = 'in_progress' and rqjl.job_id is null;
6305update replication_queue_lock set acquired = 't';
6306\""
6307`
6308[Jul 20 2023 03:30:56 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} praefect`
6309[Jul 20 2023 03:30:56 UTC (QA Tests)] INFO -- Executing: `docker unpause praefect`
6310[Jul 20 2023 03:30:56 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Running}} praefect`
6311[Jul 20 2023 03:30:56 UTC (QA Tests)] INFO -- Waiting for health check on praefect
6312[Jul 20 2023 03:30:56 UTC (QA Tests)] INFO -- Executing: `docker exec praefect gitlab-ctl status praefect`
6313[Jul 20 2023 03:30:57 UTC (QA Tests)] INFO -- ==> Retrieved a QA::Resource::Sandbox with full_path 'gitlab-qa-sandbox-group-5' via api in 0.07 seconds
6314[Jul 20 2023 03:30:57 UTC (QA Tests)] INFO -- ==> Retrieved a QA::Resource::Group with full_path 'gitlab-qa-sandbox-group-5/qa-test-2023-07-20-03-26-03-85db8ef7ea358904' via api in 0.15 seconds
6315[Jul 20 2023 03:30:59 UTC (QA Tests)] INFO -- ==> Built a QA::Resource::Project with full_path 'gitlab-qa-sandbox-group-5/qa-test-2023-07-20-03-26-03-85db8ef7ea358904/the_awesome_project-465d5847a9b88504' via api in 1.6 seconds
6316[Jul 20 2023 03:30:59 UTC (QA Tests)] INFO -- Executing: `docker exec gitlab-gitaly-cluster bash -c 'gitlab-rake "gitlab:praefect:replicas[4]"'`
6317[Jul 20 2023 03:31:29 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} postgres`
6318[Jul 20 2023 03:31:29 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT 1 as healthy_database\""
6319`
6320[Jul 20 2023 03:31:29 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly1`
6321[Jul 20 2023 03:31:29 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly2`
6322[Jul 20 2023 03:31:29 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly3`
6323[Jul 20 2023 03:31:29 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} praefect`
6324[Jul 20 2023 03:31:29 UTC (QA Tests)] INFO -- Waiting for health check on praefect
6325[Jul 20 2023 03:31:29 UTC (QA Tests)] INFO -- Executing: `docker exec praefect gitlab-ctl status praefect`
6326[Jul 20 2023 03:31:30 UTC (QA Tests)] INFO -- Waiting for health check on gitaly1
6327[Jul 20 2023 03:31:30 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly1 gitlab-ctl status gitaly`
6328[Jul 20 2023 03:31:31 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
6329`
6330[Jul 20 2023 03:31:31 UTC (QA Tests)] INFO -- Waiting for health check on gitaly2
6331[Jul 20 2023 03:31:31 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly2 gitlab-ctl status gitaly`
6332[Jul 20 2023 03:31:32 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
6333`
6334[Jul 20 2023 03:31:32 UTC (QA Tests)] INFO -- Waiting for health check on gitaly3
6335[Jul 20 2023 03:31:32 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly3 gitlab-ctl status gitaly`
6336[Jul 20 2023 03:31:33 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""
6337`
6338[Jul 20 2023 03:31:33 UTC (QA Tests)] INFO -- Clearing the replication queue
6339[Jul 20 2023 03:31:33 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"delete from replication_queue_job_lock;
6340delete from replication_queue_lock;
6341delete from replication_queue;
6342\""
6343`
6344 allows replication of different repository after interruption
6345Systems
6346 Gitaly distributed reads
6347[Jul 20 2023 03:31:33 UTC (QA Tests)] INFO -- Starting test: Systems Gitaly distributed reads reads from each node
6348[Jul 20 2023 03:31:33 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} postgres`
6349[Jul 20 2023 03:31:33 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT 1 as healthy_database\""
6350`
6351[Jul 20 2023 03:31:33 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly1`
6352[Jul 20 2023 03:31:33 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly2`
6353[Jul 20 2023 03:31:34 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly3`
6354[Jul 20 2023 03:31:34 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} praefect`
6355[Jul 20 2023 03:31:34 UTC (QA Tests)] INFO -- Waiting for health check on praefect
6356[Jul 20 2023 03:31:34 UTC (QA Tests)] INFO -- Executing: `docker exec praefect gitlab-ctl status praefect`
6357[Jul 20 2023 03:31:34 UTC (QA Tests)] INFO -- Waiting for health check on gitaly1
6358[Jul 20 2023 03:31:34 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly1 gitlab-ctl status gitaly`
6359[Jul 20 2023 03:31:35 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
6360`
6361[Jul 20 2023 03:31:36 UTC (QA Tests)] INFO -- Waiting for health check on gitaly2
6362[Jul 20 2023 03:31:36 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly2 gitlab-ctl status gitaly`
6363[Jul 20 2023 03:31:36 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
6364`
6365[Jul 20 2023 03:31:36 UTC (QA Tests)] INFO -- Waiting for health check on gitaly3
6366[Jul 20 2023 03:31:36 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly3 gitlab-ctl status gitaly`
6367[Jul 20 2023 03:31:37 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""
6368`
6369[Jul 20 2023 03:31:38 UTC (QA Tests)] INFO -- ==> Retrieved a QA::Resource::Sandbox with full_path 'gitlab-qa-sandbox-group-5' via api in 0.09 seconds
6370[Jul 20 2023 03:31:38 UTC (QA Tests)] INFO -- ==> Retrieved a QA::Resource::Group with full_path 'gitlab-qa-sandbox-group-5/qa-test-2023-07-20-03-26-03-85db8ef7ea358904' via api in 0.16 seconds
6371[Jul 20 2023 03:31:39 UTC (QA Tests)] INFO -- ==> Built a QA::Resource::Project with full_path 'gitlab-qa-sandbox-group-5/qa-test-2023-07-20-03-26-03-85db8ef7ea358904/gitaly_cluster-9dc2bbea68f77be5' via api in 1.09 seconds
6372[Jul 20 2023 03:31:39 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue where state = 'ready' or state = 'in_progress';\""
6373`
6374[Jul 20 2023 03:31:39 UTC (QA Tests)] INFO -- Executing: `docker exec gitlab-gitaly-cluster bash -c 'gitlab-rake "gitlab:praefect:replicas[5]"'`
6375[Jul 20 2023 03:32:08 UTC (QA Tests)] INFO -- Executing: `docker exec gitlab-gitaly-cluster bash -c 'curl -s http://localhost:9090/api/v1/query?query=gitaly_praefect_read_distribution'`
6376[Jul 20 2023 03:32:08 UTC (QA Tests)] INFO -- Reading from the repository
6377[Jul 20 2023 03:32:08 UTC (QA Tests)] INFO -- Executing: `docker exec gitlab-gitaly-cluster bash -c 'curl -s http://localhost:9090/api/v1/query?query=gitaly_praefect_read_distribution'`
6378[Jul 20 2023 03:32:09 UTC (QA Tests)] INFO -- Executing: `docker exec gitlab-gitaly-cluster bash -c 'curl -s http://localhost:9090/api/v1/query?query=gitaly_praefect_read_distribution'`
6379 reads from each node
6380 when a node is unhealthy
6381[Jul 20 2023 03:32:09 UTC (QA Tests)] INFO -- Starting test: Systems Gitaly distributed reads when a node is unhealthy does not read from the unhealthy node
6382[Jul 20 2023 03:32:09 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} postgres`
6383[Jul 20 2023 03:32:09 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT 1 as healthy_database\""
6384`
6385[Jul 20 2023 03:32:09 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly1`
6386[Jul 20 2023 03:32:09 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly2`
6387[Jul 20 2023 03:32:09 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly3`
6388[Jul 20 2023 03:32:09 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} praefect`
6389[Jul 20 2023 03:32:09 UTC (QA Tests)] INFO -- Waiting for health check on praefect
6390[Jul 20 2023 03:32:09 UTC (QA Tests)] INFO -- Executing: `docker exec praefect gitlab-ctl status praefect`
6391[Jul 20 2023 03:32:10 UTC (QA Tests)] INFO -- Waiting for health check on gitaly1
6392[Jul 20 2023 03:32:10 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly1 gitlab-ctl status gitaly`
6393[Jul 20 2023 03:32:11 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly1';\""
6394`
6395[Jul 20 2023 03:32:11 UTC (QA Tests)] INFO -- Waiting for health check on gitaly2
6396[Jul 20 2023 03:32:11 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly2 gitlab-ctl status gitaly`
6397[Jul 20 2023 03:32:12 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
6398`
6399[Jul 20 2023 03:32:12 UTC (QA Tests)] INFO -- Waiting for health check on gitaly3
6400[Jul 20 2023 03:32:12 UTC (QA Tests)] INFO -- Executing: `docker exec gitaly3 gitlab-ctl status gitaly`
6401[Jul 20 2023 03:32:13 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly3';\""
6402`
6403[Jul 20 2023 03:32:13 UTC (QA Tests)] INFO -- ==> Retrieved a QA::Resource::Sandbox with full_path 'gitlab-qa-sandbox-group-5' via api in 0.08 seconds
6404[Jul 20 2023 03:32:13 UTC (QA Tests)] INFO -- ==> Retrieved a QA::Resource::Group with full_path 'gitlab-qa-sandbox-group-5/qa-test-2023-07-20-03-26-03-85db8ef7ea358904' via api in 0.15 seconds
6405[Jul 20 2023 03:32:14 UTC (QA Tests)] INFO -- ==> Built a QA::Resource::Project with full_path 'gitlab-qa-sandbox-group-5/qa-test-2023-07-20-03-26-03-85db8ef7ea358904/gitaly_cluster-da2f86646812f330' via api in 1.0 seconds
6406[Jul 20 2023 03:32:14 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"select count(*) from replication_queue where state = 'ready' or state = 'in_progress';\""
6407`
6408[Jul 20 2023 03:32:14 UTC (QA Tests)] INFO -- Executing: `docker exec gitlab-gitaly-cluster bash -c 'gitlab-rake "gitlab:praefect:replicas[6]"'`
6409[Jul 20 2023 03:32:43 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly2`
6410[Jul 20 2023 03:32:43 UTC (QA Tests)] INFO -- Executing: `docker pause gitaly2`
6411[Jul 20 2023 03:32:43 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
6412`
6413[Jul 20 2023 03:32:45 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
6414`
6415[Jul 20 2023 03:32:46 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
6416`
6417[Jul 20 2023 03:32:47 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
6418`
6419[Jul 20 2023 03:32:48 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
6420`
6421[Jul 20 2023 03:32:49 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
6422`
6423[Jul 20 2023 03:32:50 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
6424`
6425[Jul 20 2023 03:32:52 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
6426`
6427[Jul 20 2023 03:32:53 UTC (QA Tests)] INFO -- Executing: `docker exec --env PGPASSWORD=SQL_PASSWORD postgres bash -c "psql -U postgres -d praefect_production -h postgres.test -c \"SELECT count(*) FROM healthy_storages WHERE storage = 'gitaly2';\""
6428`
6429[Jul 20 2023 03:32:53 UTC (QA Tests)] INFO -- Executing: `docker exec gitlab-gitaly-cluster bash -c 'curl -s http://localhost:9090/api/v1/query?query=gitaly_praefect_read_distribution'`
6430[Jul 20 2023 03:32:53 UTC (QA Tests)] INFO -- Reading from the repository
6431[Jul 20 2023 03:32:57 UTC (QA Tests)] INFO -- Executing: `docker exec gitlab-gitaly-cluster bash -c 'curl -s http://localhost:9090/api/v1/query?query=gitaly_praefect_read_distribution'`
6432[Jul 20 2023 03:32:57 UTC (QA Tests)] INFO -- Executing: `docker exec gitlab-gitaly-cluster bash -c 'curl -s http://localhost:9090/api/v1/query?query=gitaly_praefect_read_distribution'`
6433[Jul 20 2023 03:32:57 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Status}} gitaly2`
6434[Jul 20 2023 03:32:58 UTC (QA Tests)] INFO -- Executing: `docker unpause gitaly2`
6435[Jul 20 2023 03:32:58 UTC (QA Tests)] INFO -- Executing: `docker inspect -f {{.State.Running}} gitaly2`
6436 does not read from the unhealthy node
6437Finished in 7 minutes 0 seconds (files took 2.91 seconds to load)
64386 examples, 0 failures
6439Randomized with seed 48179
6440[Jul 20 2023 03:32:58 UTC (Gitlab QA)] INFO -- Shell command: `docker exec gitlab-gitaly-cluster bash -c "gitlab-psql -c 'select n_live_tup, n_dead_tup, relname from pg_stat_all_tables order by n_live_tup DESC, n_dead_tup DESC;'"`
6441[Jul 20 2023 03:32:58 UTC (Gitlab QA)] INFO -- Shell command: `docker exec gitlab-gitaly-cluster bash -c "gitlab-psql -c 'select * from pg_stat_user_tables;'"`
6442[Jul 20 2023 03:32:58 UTC (Gitlab QA)] INFO -- Shell command: `docker ps -f name=gitlab-gitaly-cluster`
6443[Jul 20 2023 03:32:59 UTC (Gitlab QA)] INFO -- Shell command: `docker rm -f gitlab-gitaly-cluster`
6444 Running after_script 00:09
6445Running after script...
6446$ if [ "$CI_JOB_STATUS" == "failed" ]; then # collapsed multi-line command
6447$ export BUNDLE_PREFIX=$([ "$RUN_WITH_BUNDLE" != "true" ] || echo "bundle exec")
6448$ echo -e "\e[0Ksection_start:`date +%s`:report_results_section[collapsed=true]\r\e[0KReport results"
6449 Report results 00:08
6471 Saving cache for successful job 00:01
6472Not uploading cache qa-e2e-ruby-3.0-25fc54ff225c814107db3bf4a8c26e7a-16 due to policy
6473 Uploading artifacts for successful job 00:03
6474Uploading artifacts...
6475gitlab-qa-run-*: found 263 matching artifact files and directories
6476Uploading artifacts as "archive" to coordinator... 201 Created id=4694998482 responseStatus=201 Created token=64_qg_sn
6477Uploading artifacts...
6478gitlab-qa-run-*/**/rspec-*.xml: found 1 matching artifact files and directories
6479Uploading artifacts as "junit" to coordinator... 201 Created id=4694998482 responseStatus=201 Created token=64_qg_sn
6480Uploading artifacts...
6481/builds/gitlab-org/gitlab/suite_status.env: found 1 matching artifact files and directories
6482Uploading artifacts as "dotenv" to coordinator... 201 Created id=4694998482 responseStatus=201 Created token=64_qg_sn
6483 Cleaning up project directory and file based variables 00:00
6484Job succeeded