Skip to content

Extend gstg watch to include NameError

See gitlab-org/gitlab#348698 (closed), and gitlab-com/gl-infra/production#6083 (closed)

Testing

I modified elastic/managed-objects/nonprod-log/watches/no_method_error.jsonnet to extend the range:

diff --git a/elastic/managed-objects/nonprod-log/watches/no_method_error.jsonnet b/elastic/managed-objects/nonprod-log/watches/no_method_error.jsonnet
index ac38a962..27ab33fa 100644
--- a/elastic/managed-objects/nonprod-log/watches/no_method_error.jsonnet
+++ b/elastic/managed-objects/nonprod-log/watches/no_method_error.jsonnet
@@ -16,7 +16,7 @@ local es_query = {
           {
             range: {
               '@timestamp': {
-                gte: std.format('now-%dm', query_period),
+                gte: std.format('now-%dd', query_period),
                 lte: 'now',
               },
             },

Then generated the JSON with:

jsonnet elastic/managed-objects/nonprod-log/watches/no_method_error.jsonnet

Simulating the watch in Elastic gave me:

    "condition": {
      "type": "compare",
      "status": "success",
      "met": true,
      "compare": {
        "resolved_values": {
          "ctx.payload.hits.total": 71
        }
      }
    },
    "actions": [
      {
        "id": "notify-slack",
        "type": "slack",
        "status": "simulated",
        "slack": {
          "message": {
            "from": "ElasticCloud Watcher: NoMethodError",
            "to": [
              "#staging"
            ],
            "text": "NoMethodError: 71 errors detected!",
            "attachments": [
              {
                "title": ":spiral_note_pad: NoMethodError in Rails logs:",
                "text": "https://nonprod-log.gitlab.net/goto/28345690-61e0-11ec-aec7-f387696fcb42"
              },
              {
                "title": ":spiral_note_pad: NoMethodError in Sidekiq logs:",
                "text": "https://nonprod-log.gitlab.net/goto/3e0b0180-61e0-11ec-aec7-f387696fcb42"
              },
              {
                "title": ":runbooks: Runbook:",
                "text": "https://gitlab.com/gitlab-com/runbooks/-/blob/master/docs/uncategorized/staging-environment.md#elasticcloud-watcher-nomethoderror"
              }
            ]
          }
        }
      }
    ]
  },
  "messages": []
}

Which matches what I see in https://nonprod-log.gitlab.net/goto/28345690-61e0-11ec-aec7-f387696fcb42

Screen_Shot_2021-12-21_at_11.29.49_AM

Edited by Thong Kuah

Merge request reports

Loading