CI: Print most recent log errors
What does this merge request do and why?
This makes inspect logfile artifacts less likely and spotting errors easier.
Contributes to #3037 (closed).
How to set up and validate locally
gdk rake gdk:recent_log_errorsMAX_LINES=10 gdk rake gdk:recent_log_errors
$ gdk rake gdk:recent_log_errors
Searching for errors in 45 logfiles...
-------------------- START gitlab/log/application_json.log ---------------------
{"severity":"ERROR","time":"2025-10-13T07:36:27.344Z","message":"Projects::CreateService import_project failed to be reported to collector at 127.0.0.1:9091"}
{"severity":"ERROR","time":"2025-10-13T07:36:27.344Z","message":"Projects::CreateService import_project failed to be reported to collector at 127.0.0.1:9091"}
{"severity":"ERROR","time":"2025-10-13T07:36:27.344Z","message":"Projects::CreateService import_project failed to be reported to collector at 127.0.0.1:9091"}
{"severity":"ERROR","time":"2025-10-13T07:36:27.344Z","message":"Projects::CreateService import_project failed to be reported to collector at 127.0.0.1:9091"}
{"severity":"ERROR","time":"2025-10-13T07:36:27.344Z","message":"Projects::CreateService import_project failed to be reported to collector at 127.0.0.1:9091"}
[...]
------------------- START log/rails-background-jobs/current --------------------
2025-10-13_07:49:43.52497 rails-background-jobs : E, [2025-10-13T09:49:43.524808 #848657] ERROR -- : Exception rescued in listen-run_thread:
2025-10-13_07:49:43.52498 rails-background-jobs : INotify::QueueOverflowError: inotify event queue has overflowed.
2025-10-13_07:49:43.52502 rails-background-jobs : --- Thread.new ---
2025-10-13_11:43:54.02734 rails-background-jobs : from /home/peter/mise/installs/ruby/3.3.9/lib/ruby/3.3.0/bundled_gems.rb:69:in `block (2 levels) in replace_require'
2025-10-13_11:43:54.03047 rails-background-jobs : from /home/peter/mise/installs/ruby/3.3.9/lib/ruby/3.3.0/bundled_gems.rb:69:in `block (2 levels) in replace_require'
... 342 lines omitted. See log/rails-background-jobs/current for the full log.
-------------------- END log/rails-background-jobs/current ---------------------
------------------------- START log/rails-web/current --------------------------
2025-10-13_07:49:43.74026 rails-web : INotify::QueueOverflowError: inotify event queue has overflowed.
2025-10-13_07:49:43.85544 rails-web : INotify::QueueOverflowError: inotify event queue has overflowed.
2025-10-13_07:49:45.42415 rails-web : INotify::QueueOverflowError: inotify event queue has overflowed.
2025-10-13_16:52:56.42633 rails-web : {"timestamp":"2025-10-13T16:52:56.426Z","pid":848404,"message":"! Terminating timed out worker (Worker 0 failed to check in within 60 seconds): 853590"}
2025-10-14_08:03:16.16947 rails-web : {"timestamp":"2025-10-14T08:03:16.169Z","pid":848404,"message":"! Terminating timed out worker (Worker 1 failed to check in within 60 seconds): 853594"}
... 521 lines omitted. See log/rails-web/current for the full log.
-------------------------- END log/rails-web/current ---------------------------
$ gdk rake gdk:recent_log_errors | grep START
-------------------- START gitlab/log/application_json.log ---------------------
----------------------- START gitlab/log/development.log -----------------------
--------------------- START gitlab/log/exceptions_json.log ---------------------
----------------------- START gitlab/log/gitaly-test.log -----------------------
---------------------- START gitlab/log/gitaly2-test.log -----------------------
---------------------- START gitlab/log/praefect-test.log ----------------------
------------------------- START gitlab/log/sidekiq.log -------------------------
-------------------------- START gitlab/log/test.log ---------------------------
---------------- START log/clickhouse/clickhouse-server.err.log ----------------
------------------ START log/clickhouse/clickhouse-server.log ------------------
------------------------- START log/clickhouse/current -------------------------
------------------ START log/gitlab-topology-service/current -------------------
-------------------------- START log/openbao/current ---------------------------
------------------------ START log/openbao/service.log -------------------------
------------------------- START log/postgresql/current -------------------------
--------------------- START log/praefect-gitaly-0/current ----------------------
-------------------------- START log/praefect/current --------------------------
------------------- START log/rails-background-jobs/current --------------------
------------------------- START log/rails-web/current --------------------------
Impacted categories
The following categories relate to this merge request:
-
gdk-reliability - e.g. When a GDK action fails to complete. -
gdk-usability - e.g. Improvements or suggestions around how the GDK functions. -
gdk-performance - e.g. When a GDK action is slow or times out.
Merge request checklist
-
This MR references an issue describing the change. -
This change is backward compatible. If not, include steps to communicate to users. -
Tests added for new functionality. If not, raise an issue to follow-up. -
Observability added/updated (logging, metrics, tracing). -
Documentation added/updated. -
Announcement added for notable changes. -
gdk doctortest added.
Edited by Peter Leitzen