PagerDuty Webhook integration doesn't seem to be working
Summary
Customers are complaining about PagerDuty integration not creating incidents under the associated projects' Monitoring -> Incidents
Steps to reproduce
Example Project
Example Project has been identified under ZD343219 (internal link) Yet another affected projects can be found through Kibana (when querying for 'json.path: "/incidents/integrations/pagerduty"' in the rails logs).
What is the current bug behavior?
PagerDuty contacts the defined WebHook endpoints (<namespace>/-/incidents/integrations/pagerduty),
GitLab rails returnes 202 status, but no incidents/issues are created in GitLab.
What is the expected correct behavior?
Issues/Incidents should be created in GitLab as per the integration's description under Create incidents via the PagerDuty webhook
Relevant logs and/or screenshots
Identified projects are beeing actively contacted by PagerDuty, yet no new incidents are created automatically. Configuration on GitLab side seems to be correct. Cannot verify the correctness of PagerDuty setup, but incoming traffic traffic (from PD) can be seen, thus we may expect that part to be correct as well.
In the logs I have found incoming requests from PD like these:
Click to expand
{
"_index": "pubsub-rails-inf-gprd-014358",
"_type": "_doc",
"_id": "KWmGgYQByAWNNFiR0aXc",
"_version": 1,
"_score": 1,
"_ignored": [
"json.params.value.keyword"
],
"_source": {
"@timestamp": "2022-11-16T17:40:06.660Z",
"host": {
"name": "pubsubbeat-pubsub-rails-inf-gprd-5f5c54989f-ltcwv"
},
"publish_time": "2022-11-16T17:40:06.533Z",
"kubernetes": {
"pod_ip": "10.66.52.182",
"container_image": "dev.gitlab.org:5005/gitlab/charts/components/images/gitlab-webservice-ee:15-6-202211160920-0f412e1c015",
"host": "gke-gprd-us-east1-c-generic-1-d5dc3c5d-ffdd",
"container_name": "webservice",
"namespace_name": "gitlab",
"region": "us-east1-c",
"pod_name": "gitlab-webservice-web-84ffbf6b7b-qqfj8"
},
"json": {
"db_write_count": 0,
"mem_total_bytes": 1424208,
"mem_bytes": 793008,
"tier": "sv",
"rack_attack_redis_count": 1,
"worker_id": "puma_0",
"stage": "main",
"type": "web",
"db_main_wal_count": 0,
"db_main_replica_cached_count": 1,
"db_ci_replica_wal_cached_count": 0,
"meta.feature_category": "incident_management",
"db_ci_replica_duration_s": 0,
"controller": "Projects::IncidentManagement::PagerDutyIncidentsController",
"redis_rate_limiting_duration_s": 0.00103,
"db_replica_cached_count": 1,
"meta.remote_ip": "18.194.177.30",
"correlation_id": "01GJ0RDDMT726Z7BMKQH7K5H2Q",
"db_replica_wal_cached_count": 0,
"meta.caller_id": "Projects::IncidentManagement::PagerDutyIncidentsController#create",
"view_duration_s": 0,
"path": "<UNDISCLOSED>/-/incidents/integrations/pagerduty",
"redis_calls": 3,
"db_main_replica_wal_cached_count": 0,
"db_main_duration_s": 0,
"db_cached_count": 1,
"db_primary_cached_count": 0,
"db_main_replica_wal_count": 0,
"status": 202,
"db_ci_replica_count": 0,
"db_ci_count": 0,
"db_count": 3,
"redis_duration_s": 0.00103,
"db_ci_replica_wal_count": 0,
"cpu_s": 0.040922,
"rate_limiting_gates": [],
"db_replica_duration_s": 0.005,
"duration_s": 0.01645,
"redis_read_bytes": 4,
"time": "2022-11-16T17:40:00.329Z",
"tag": "web-rails.var.log.containers.gitlab-webservice-web-84ffbf6b7b-qqfj8_gitlab_webservice-5821cd34a064dabf525ef6262b701806fb8143415785a0577f35dd494c262f3c.log",
"db_duration_s": 0.00284,
"meta.root_namespace": "<UNDISCLOSED>",
"response_bytes": 0,
"db_primary_count": 0,
"db_ci_duration_s": 0,
"component": "gitlab",
"format": "json",
"db_main_replica_count": 3,
"db_ci_wal_cached_count": 0,
"db_primary_wal_cached_count": 0,
"action": "create",
"db_ci_wal_count": 0,
"mem_objects": 15780,
"remote_ip": "18.194.177.30",
"db_ci_cached_count": 0,
"environment": "gprd",
"level": "info",
"subcomponent": "production_json",
"db_main_replica_duration_s": 0.005,
"pid": 53,
"db_replica_count": 3,
"rack_attack_redis_duration_s": 0.0013403799966908991,
"db_main_cached_count": 0,
"request_urgency": "low",
"redis_rate_limiting_read_bytes": 4,
"db_main_wal_cached_count": 0,
"redis_write_bytes": 249,
"params": [
{
"key": "event",
"value": "{\"id\"=>\"01DCQEF6PPXWHGXQXV5D2FFBJF\", \"event_type\"=>\"incident.acknowledged\", \"resource_type\"=>\"incident\", \"occurred_at\"=>\"2022-11-16T17:39:59.577Z\", \"agent\"=>{\"html_url\"=>\"https://<UNDISCLOSED>/users/<UNDISCLOSED>\", \"id\"=>\"PSNIFEK\", \"self\"=>\"https://api.eu.pagerduty.com/users/<UNDISCLOSED>\", \"summary\"=>\"<UNDISCLOSED>\", \"type\"=>\"user_reference\"}, \"client\"=>nil, \"data\"=>{\"id\"=>\"Q31Q9L1D1FUCKQ\", \"type\"=>\"incident\", \"self\"=>\"https://api.eu.pagerduty.com/incidents/Q31Q9L1D1FUCKQ\", \"html_url\"=>\"https://<UNDISCLOSED>/incidents/Q31Q9L1D1FUCKQ\", \"number\"=>661, \"status\"=>\"acknowledged\", \"incident_key\"=>\"[FILTERED]\", \"created_at\"=>\"2022-11-16T17:39:00Z\", \"title\"=>\"[FILTERED]\", \"service\"=>{\"html_url\"=>\"https://<UNDISCLOSED>/services/<UNDISCLOSED>\", \"id\"=>\"<UNDISCLOSED>\", \"self\"=>\"https://api.eu.pagerduty.com/services/<UNDISCLOSED>\", \"summary\"=>\"GIPCDP - Customer Services\", \"type\"=>\"service_reference\"}, \"assignees\"=>[{\"html_url\"=>\"https://<UNDISCLOSED>/users/<UNDISCLOSED>\", \"id\"=>\"<UNDISCLOSED>\", \"self\"=>\"https://api.eu.pagerduty.com/users/<UNDISCLOSED>\", \"summary\"=>\"<UNDISCLOSED>\", \"type\"=>\"user_reference\"}], \"escalation_policy\"=>{\"html_url\"=>\"https://<UNDISCLOSED>/escalation_policies/<UNDISCLOSED>\", \"id\"=>\"<UNDISCLOSED>\", \"self\"=>\"https://api.eu.pagerduty.com/escalation_policies/<UNDISCLOSED>\", \"summary\"=>\"Standard Incident Escalation Policy\", \"type\"=>\"escalation_policy_reference\"}, \"teams\"=>[{\"html_url\"=>\"https://<UNDISCLOSED>/teams/<UNDISCLOSED>\", \"id\"=>\"<UNDISCLOSED>\", \"self\"=>\"https://api.eu.pagerduty.com/teams/<UNDISCLOSED>\", \"summary\"=>\"<UNDISCLOSED> - Team\", \"type\"=>\"team_reference\"}], \"priority\"=>nil, \"urgency\"=>\"high\", \"conference_bridge\"=>nil, \"resolve_reason\"=>nil}}"
},
{
"value": "[FILTERED]",
"key": "token"
},
{
"key": "namespace_id",
"value": "<UNDISCLOSED>"
},
{
"key": "project_id",
"value": "<UNDISCLOSED>"
},
{
"value": "{\"event\"=>{\"id\"=>\"01DCQEF6PPXWHGXQXV5D2FFBJF\", \"event_type\"=>\"incident.acknowledged\", \"resource_type\"=>\"incident\", \"occurred_at\"=>\"2022-11-16T17:39:59.577Z\", \"agent\"=>{\"html_url\"=>\"https://<UNDISCLOSED>/users/<UNDISCLOSED>\", \"id\"=>\"<UNDISCLOSED>\", \"self\"=>\"https://api.eu.pagerduty.com/users/<UNDISCLOSED>\", \"summary\"=>\"<UNDISCLOSED>\", \"type\"=>\"user_reference\"}, \"client\"=>nil, \"data\"=>{\"id\"=>\"Q31Q9L1D1FUCKQ\", \"type\"=>\"incident\", \"self\"=>\"https://api.eu.pagerduty.com/incidents/Q31Q9L1D1FUCKQ\", \"html_url\"=>\"https://<UNDISCLOSED>/incidents/Q31Q9L1D1FUCKQ\", \"number\"=>661, \"status\"=>\"acknowledged\", \"incident_key\"=>\"[FILTERED]\", \"created_at\"=>\"2022-11-16T17:39:00Z\", \"title\"=>\"[FILTERED]\", \"service\"=>{\"html_url\"=>\"https://<UNDISCLOSED>/services/<UNDISCLOSED>\", \"id\"=>\"<UNDISCLOSED>\", \"self\"=>\"https://api.eu.pagerduty.com/services/<UNDISCLOSED>\", \"summary\"=>\"<UNDISCLOSED> - Customer Services\", \"type\"=>\"service_reference\"}, \"assignees\"=>[{\"html_url\"=>\"https://<UNDISCLOSED>/users/PSNIFEK\", \"id\"=>\"<UNDISCLOSED>\", \"self\"=>\"https://api.eu.pagerduty.com/users/<UNDISCLOSED>\", \"summary\"=>\"<UNDISCLOSED>\", \"type\"=>\"user_reference\"}], \"escalation_policy\"=>{\"html_url\"=>\"https://<UNDISCLOSED>/escalation_policies/<UNDISCLOSED>\", \"id\"=>\"<UNDISCLOSED>\", \"self\"=>\"https://api.eu.pagerduty.com/escalation_policies/<UNDISCLOSED>\", \"summary\"=>\"Standard Incident Escalation Policy\", \"type\"=>\"escalation_policy_reference\"}, \"teams\"=>[{\"html_url\"=>\"https://<UNDISCLOSED>/teams/<UNDISCLOSED>\", \"id\"=>\"<UNDISCLOSED>\", \"self\"=>\"https://api.eu.pagerduty.com/teams/<UNDISCLOSED>\", \"summary\"=>\"<UNDISCLOSED> - Team\", \"type\"=>\"team_reference\"}], \"priority\"=>nil, \"urgency\"=>\"high\", \"conference_bridge\"=>nil, \"resolve_reason\"=>nil}}}",
"key": "pager_duty_incident"
}
],
"db_primary_duration_s": 0,
"meta.client_id": "ip/18.194.177.30",
"ua": "PagerDuty-Webhook/V3.0",
"target_duration_s": 5,
"queue_duration_s": 0.012583,
"db_ci_replica_cached_count": 0,
"db_primary_wal_count": 0,
"shard": "default",
"redis_rate_limiting_calls": 3,
"method": "POST",
"meta.project": "<UNDISCLOSED>",
"db_replica_wal_count": 0,
"redis_rate_limiting_write_bytes": 249,
"cf_ray": "76b2025cf93cbb7f-IAD",
"db_main_count": 0,
"mem_mallocs": 2596
},
"type": "pubsubbeat-pubsub-rails-inf-gprd-5f5c54989f-ltcwv"
},
"fields": {
"json.type.keyword": [
"web"
],
"json.db_replica_wal_cached_count": [
0
],
"json.mem_mallocs": [
2596
],
"json.redis_duration_s": [
0.00103
],
"host.name.keyword": [
"pubsubbeat-pubsub-rails-inf-gprd-5f5c54989f-ltcwv"
],
"json.db_primary_cached_count": [
0
],
"json.params.key": [
"event",
"token",
"namespace_id",
"project_id",
"pager_duty_incident"
],
"json.db_replica_count": [
3
],
"json.mem_total_bytes": [
1424208
],
"type": [
"pubsubbeat-pubsub-rails-inf-gprd-5f5c54989f-ltcwv"
],
"json.db_main_wal_count": [
0
],
"json.mem_bytes": [
793008
],
"publish_time": [
"2022-11-16T17:40:06.533Z"
],
"json.db_primary_wal_cached_count": [
0
],
"json.params.value": [
"{\"id\"=>\"01DCQEF6PPXWHGXQXV5D2FFBJF\", \"event_type\"=>\"incident.acknowledged\", \"resource_type\"=>\"incident\", \"occurred_at\"=>\"2022-11-16T17:39:59.577Z\", \"agent\"=>{\"html_url\"=>\"https://<UNDISCLOSED>/users/<UNDISCLOSED>\", \"id\"=>\"<UNDISCLOSED>\", \"self\"=>\"https://api.eu.pagerduty.com/users/<UNDISCLOSED>\", \"summary\"=>\"<UNDISCLOSED>\", \"type\"=>\"user_reference\"}, \"client\"=>nil, \"data\"=>{\"id\"=>\"Q31Q9L1D1FUCKQ\", \"type\"=>\"incident\", \"self\"=>\"https://api.eu.pagerduty.com/incidents/Q31Q9L1D1FUCKQ\", \"html_url\"=>\"https://<UNDISCLOSED>/incidents/Q31Q9L1D1FUCKQ\", \"number\"=>661, \"status\"=>\"acknowledged\", \"incident_key\"=>\"[FILTERED]\", \"created_at\"=>\"2022-11-16T17:39:00Z\", \"title\"=>\"[FILTERED]\", \"service\"=>{\"html_url\"=>\"https://<UNDISCLOSED>/services/<UNDISCLOSED>\", \"id\"=>\"<UNDISCLOSED>\", \"self\"=>\"https://api.eu.pagerduty.com/services/<UNDISCLOSED>\", \"summary\"=>\"<UNDISCLOSED> - Customer Services\", \"type\"=>\"service_reference\"}, \"assignees\"=>[{\"html_url\"=>\"https://<UNDISCLOSED>/users/PSNIFEK\", \"id\"=>\"PSNIFEK\", \"self\"=>\"https://api.eu.pagerduty.com/users/PSNIFEK\", \"summary\"=>\"<UNDISCLOSED>\", \"type\"=>\"user_reference\"}], \"escalation_policy\"=>{\"html_url\"=>\"https://gi-cloud.eu.pagerduty.com/escalation_policies/PT0YFSR\", \"id\"=>\"PT0YFSR\", \"self\"=>\"https://api.eu.pagerduty.com/escalation_policies/<UNDISCLOSED>\", \"summary\"=>\"Standard Incident Escalation Policy\", \"type\"=>\"escalation_policy_reference\"}, \"teams\"=>[{\"html_url\"=>\"https://<UNDISCLOSED>.eu.pagerduty.com/teams/<UNDISCLOSED>\", \"id\"=>\"<UNDISCLOSED>\", \"self\"=>\"https://api.eu.pagerduty.com/teams/<UNDISCLOSED>\", \"summary\"=>\"<UNDISCLOSED> - Team\", \"type\"=>\"team_reference\"}], \"priority\"=>nil, \"urgency\"=>\"high\", \"conference_bridge\"=>nil, \"resolve_reason\"=>nil}}",
"[FILTERED]",
"<UNDISCLOSED>",
"<UNDISCLOSED>",
"{\"event\"=>{\"id\"=>\"01DCQEF6PPXWHGXQXV5D2FFBJF\", \"event_type\"=>\"incident.acknowledged\", \"resource_type\"=>\"incident\", \"occurred_at\"=>\"2022-11-16T17:39:59.577Z\", \"agent\"=>{\"html_url\"=>\"https://<UNDISCLOSED>.eu.pagerduty.com/users/<UNDISCLOSED>\", \"id\"=>\"<UNDISCLOSED>\", \"self\"=>\"https://api.eu.pagerduty.com/users/<UNDISCLOSED>\", \"summary\"=>\"<UNDISCLOSED>\", \"type\"=>\"user_reference\"}, \"client\"=>nil, \"data\"=>{\"id\"=>\"Q31Q9L1D1FUCKQ\", \"type\"=>\"incident\", \"self\"=>\"https://api.eu.pagerduty.com/incidents/Q31Q9L1D1FUCKQ\", \"html_url\"=>\"https://<UNDISCLOSED>.eu.pagerduty.com/incidents/Q31Q9L1D1FUCKQ\", \"number\"=>661, \"status\"=>\"acknowledged\", \"incident_key\"=>\"[FILTERED]\", \"created_at\"=>\"2022-11-16T17:39:00Z\", \"title\"=>\"[FILTERED]\", \"service\"=>{\"html_url\"=>\"https://<UNDISCLOSED>.eu.pagerduty.com/services/P6VSFXU\", \"id\"=>\"<UNDISCLOSED>\", \"self\"=>\"https://api.eu.pagerduty.com/services/<UNDISCLOSED>\", \"summary\"=>\"<UNDISCLOSED> - Customer Services\", \"type\"=>\"service_reference\"}, \"assignees\"=>[{\"html_url\"=>\"https://<UNDISCLOSED>.eu.pagerduty.com/users/<UNDISCLOSED>\", \"id\"=>\"<UNDISCLOSED>\", \"self\"=>\"https://api.eu.pagerduty.com/users/PSNIFEK\", \"summary\"=>\"<UNDISCLOSED>\", \"type\"=>\"user_reference\"}], \"escalation_policy\"=>{\"html_url\"=>\"https://<UNDISCLOSED>.eu.pagerduty.com/escalation_policies/<UNDISCLOSED>\", \"id\"=>\"<UNDISCLOSED>\", \"self\"=>\"https://api.eu.pagerduty.com/escalation_policies/<UNDISCLOSED>\", \"summary\"=>\"Standard Incident Escalation Policy\", \"type\"=>\"escalation_policy_reference\"}, \"teams\"=>[{\"html_url\"=>\"https://<UNDISCLOSED>.eu.pagerduty.com/teams/<UNDISCLOSED>\", \"id\"=>\"<UNDISCLOSED>\", \"self\"=>\"https://api.eu.pagerduty.com/teams/<UNDISCLOSED>\", \"summary\"=>\"<UNDISCLOSED> - Team\", \"type\"=>\"team_reference\"}], \"priority\"=>nil, \"urgency\"=>\"high\", \"conference_bridge\"=>nil, \"resolve_reason\"=>nil}}}"
],
"json.cf_ray": [
"76b2025cf93cbb7f-IAD"
],
"json.target_duration_s": [
5
],
"json.meta.caller_id": [
"Projects::IncidentManagement::PagerDutyIncidentsController#create"
],
"json.meta.root_namespace.keyword": [
"<UNDISCLOSED>"
],
"json.subcomponent.keyword": [
"production_json"
],
"json.db_replica_cached_count": [
1
],
"json.db_ci_replica_count": [
0
],
"json.db_ci_cached_count": [
0
],
"json.request_urgency": [
"low"
],
"json.environment": [
"gprd"
],
"json.db_primary_duration_s": [
0
],
"json.queue_duration_s": [
0.012583
],
"json.rack_attack_redis_duration_s": [
0.0013403799
],
"json.rack_attack_redis_count": [
1
],
"kubernetes.pod_ip": [
"10.66.52.182"
],
"json.params.value.keyword": [
"[FILTERED]",
"<UNDISCLOSED>",
"<UNDISCLOSED>"
],
"json.type": [
"web"
],
"json.meta.feature_category": [
"incident_management"
],
"json.db_main_wal_cached_count": [
0
],
"json.worker_id.keyword": [
"puma_0"
],
"json.environment.keyword": [
"gprd"
],
"json.duration_s": [
0.01645
],
"json.path.keyword": [
"<UNDISCLOSED>/-/incidents/integrations/pagerduty"
],
"json.redis_read_bytes": [
4
],
"json.action": [
"create"
],
"json.level.keyword": [
"info"
],
"json.meta.feature_category.keyword": [
"incident_management"
],
"kubernetes.namespace_name.keyword": [
"gitlab"
],
"json.redis_calls": [
3
],
"json.meta.client_id.keyword": [
"ip/18.194.177.30"
],
"json.db_ci_replica_wal_count": [
0
],
"json.format": [
"json"
],
"json.db_primary_count": [
0
],
"json.db_main_replica_duration_s": [
0.005
],
"json.db_main_cached_count": [
0
],
"json.redis_rate_limiting_duration_s": [
0.00103
],
"json.ua.keyword": [
"PagerDuty-Webhook/V3.0"
],
"json.db_write_count": [
0
],
"json.redis_rate_limiting_write_bytes": [
249
],
"json.shard.keyword": [
"default"
],
"@timestamp": [
"2022-11-16T17:40:06.660Z"
],
"json.worker_id": [
"puma_0"
],
"json.db_ci_duration_s": [
0
],
"kubernetes.pod_name": [
"gitlab-webservice-web-84ffbf6b7b-qqfj8"
],
"json.db_replica_duration_s": [
0.005
],
"json.meta.remote_ip": [
"18.194.177.30"
],
"json.method.keyword": [
"POST"
],
"json.db_main_count": [
0
],
"json.db_main_replica_wal_count": [
0
],
"json.meta.project": [
"<UNDISCLOSED>"
],
"json.redis_rate_limiting_calls": [
3
],
"kubernetes.container_image.keyword": [
"dev.gitlab.org:5005/gitlab/charts/components/images/gitlab-webservice-ee:15-6-202211160920-0f412e1c015"
],
"json.db_main_replica_wal_cached_count": [
0
],
"type.keyword": [
"pubsubbeat-pubsub-rails-inf-gprd-5f5c54989f-ltcwv"
],
"kubernetes.container_image": [
"dev.gitlab.org:5005/gitlab/charts/components/images/gitlab-webservice-ee:15-6-202211160920-0f412e1c015"
],
"json.db_cached_count": [
1
],
"host.name": [
"pubsubbeat-pubsub-rails-inf-gprd-5f5c54989f-ltcwv"
],
"json.component.keyword": [
"gitlab"
],
"json.db_ci_wal_cached_count": [
0
],
"kubernetes.host.keyword": [
"gke-gprd-us-east1-c-generic-1-d5dc3c5d-ffdd"
],
"json.tier.keyword": [
"sv"
],
"json.tag": [
"web-rails.var.log.containers.gitlab-webservice-web-84ffbf6b7b-qqfj8_gitlab_webservice-5821cd34a064dabf525ef6262b701806fb8143415785a0577f35dd494c262f3c.log"
],
"json.cf_ray.keyword": [
"76b2025cf93cbb7f-IAD"
],
"kubernetes.container_name": [
"webservice"
],
"json.level": [
"info"
],
"json.path": [
"<UNDISCLOSED>/-/incidents/integrations/pagerduty"
],
"kubernetes.pod_ip.keyword": [
"10.66.52.182"
],
"json.redis_rate_limiting_read_bytes": [
4
],
"json.db_replica_wal_count": [
0
],
"json.response_bytes": [
0
],
"kubernetes.container_name.keyword": [
"webservice"
],
"json.pid": [
53
],
"json.ua": [
"PagerDuty-Webhook/V3.0"
],
"json.db_primary_wal_count": [
0
],
"json.remote_ip": [
"18.194.177.30"
],
"json.stage": [
"main"
],
"json.db_count": [
3
],
"json.db_main_replica_count": [
3
],
"kubernetes.region": [
"us-east1-c"
],
"json.request_urgency.keyword": [
"low"
],
"json.redis_write_bytes": [
249
],
"json.meta.caller_id.keyword": [
"Projects::IncidentManagement::PagerDutyIncidentsController#create"
],
"json.component": [
"gitlab"
],
"json.cpu_s": [
0.040922
],
"json.tag.keyword": [
"web-rails.var.log.containers.gitlab-webservice-web-84ffbf6b7b-qqfj8_gitlab_webservice-5821cd34a064dabf525ef6262b701806fb8143415785a0577f35dd494c262f3c.log"
],
"json.params.key.keyword": [
"event",
"token",
"namespace_id",
"project_id",
"pager_duty_incident"
],
"hour": [
17
],
"json.controller": [
"Projects::IncidentManagement::PagerDutyIncidentsController"
],
"json.meta.project.keyword": [
"<UNDISCLOSED>"
],
"json.subcomponent": [
"production_json"
],
"json.time": [
"2022-11-16T17:40:00.329Z"
],
"json.meta.root_namespace": [
"<UNDISCLOSED>"
],
"json.db_main_duration_s": [
0
],
"kubernetes.host": [
"gke-gprd-us-east1-c-generic-1-d5dc3c5d-ffdd"
],
"json.db_main_replica_cached_count": [
1
],
"kubernetes.pod_name.keyword": [
"gitlab-webservice-web-84ffbf6b7b-qqfj8"
],
"json.mem_objects": [
15780
],
"json.shard": [
"default"
],
"json.stage.keyword": [
"main"
],
"json.correlation_id.keyword": [
"01GJ0RDDMT726Z7BMKQH7K5H2Q"
],
"json.method": [
"POST"
],
"json.correlation_id": [
"01GJ0RDDMT726Z7BMKQH7K5H2Q"
],
"json.db_ci_replica_cached_count": [
0
],
"kubernetes.region.keyword": [
"us-east1-c"
],
"json.db_ci_replica_wal_cached_count": [
0
],
"json.db_ci_count": [
0
],
"json.meta.remote_ip.keyword": [
"18.194.177.30"
],
"json.db_ci_replica_duration_s": [
0
],
"json.tier": [
"sv"
],
"ua_string_base": [
"PagerDuty-Webhook"
],
"kubernetes.namespace_name": [
"gitlab"
],
"json.db_duration_s": [
0.00284
],
"json.db_ci_wal_count": [
0
],
"json.remote_ip.keyword": [
"18.194.177.30"
],
"json.status": [
202
],
"json.action.keyword": [
"create"
],
"json.format.keyword": [
"json"
],
"json.view_duration_s": [
0
],
"json.controller.keyword": [
"Projects::IncidentManagement::PagerDutyIncidentsController"
],
"json.meta.client_id": [
"ip/18.194.177.30"
]
},
"ignored_field_values": {
"json.params.value.keyword": [
"{\"id\"=>\"01DCQEF6PPXWHGXQXV5D2FFBJF\", \"event_type\"=>\"incident.acknowledged\", \"resource_type\"=>\"incident\", \"occurred_at\"=>\"2022-11-16T17:39:59.577Z\", \"agent\"=>{\"html_url\"=>\"https://gi-cloud.eu.pagerduty.com/users/PSNIFEK\", \"id\"=>\"PSNIFEK\", \"self\"=>\"https://api.eu.pagerduty.com/users/PSNIFEK\", \"summary\"=>\"Massimiliano Gualtieri\", \"type\"=>\"user_reference\"}, \"client\"=>nil, \"data\"=>{\"id\"=>\"Q31Q9L1D1FUCKQ\", \"type\"=>\"incident\", \"self\"=>\"https://api.eu.pagerduty.com/incidents/Q31Q9L1D1FUCKQ\", \"html_url\"=>\"https://gi-cloud.eu.pagerduty.com/incidents/Q31Q9L1D1FUCKQ\", \"number\"=>661, \"status\"=>\"acknowledged\", \"incident_key\"=>\"[FILTERED]\", \"created_at\"=>\"2022-11-16T17:39:00Z\", \"title\"=>\"[FILTERED]\", \"service\"=>{\"html_url\"=>\"https://gi-cloud.eu.pagerduty.com/services/P6VSFXU\", \"id\"=>\"P6VSFXU\", \"self\"=>\"https://api.eu.pagerduty.com/services/P6VSFXU\", \"summary\"=>\"GIPCDP - Customer Services\", \"type\"=>\"service_reference\"}, \"assignees\"=>[{\"html_url\"=>\"https://gi-cloud.eu.pagerduty.com/users/PSNIFEK\", \"id\"=>\"PSNIFEK\", \"self\"=>\"https://api.eu.pagerduty.com/users/PSNIFEK\", \"summary\"=>\"Massimiliano Gualtieri\", \"type\"=>\"user_reference\"}], \"escalation_policy\"=>{\"html_url\"=>\"https://gi-cloud.eu.pagerduty.com/escalation_policies/PT0YFSR\", \"id\"=>\"PT0YFSR\", \"self\"=>\"https://api.eu.pagerduty.com/escalation_policies/PT0YFSR\", \"summary\"=>\"Standard Incident Escalation Policy\", \"type\"=>\"escalation_policy_reference\"}, \"teams\"=>[{\"html_url\"=>\"https://gi-cloud.eu.pagerduty.com/teams/PG0PF4W\", \"id\"=>\"PG0PF4W\", \"self\"=>\"https://api.eu.pagerduty.com/teams/PG0PF4W\", \"summary\"=>\"GIPCDP - Team\", \"type\"=>\"team_reference\"}], \"priority\"=>nil, \"urgency\"=>\"high\", \"conference_bridge\"=>nil, \"resolve_reason\"=>nil}}",
"{\"event\"=>{\"id\"=>\"01DCQEF6PPXWHGXQXV5D2FFBJF\", \"event_type\"=>\"incident.acknowledged\", \"resource_type\"=>\"incident\", \"occurred_at\"=>\"2022-11-16T17:39:59.577Z\", \"agent\"=>{\"html_url\"=>\"https://gi-cloud.eu.pagerduty.com/users/PSNIFEK\", \"id\"=>\"PSNIFEK\", \"self\"=>\"https://api.eu.pagerduty.com/users/PSNIFEK\", \"summary\"=>\"Massimiliano Gualtieri\", \"type\"=>\"user_reference\"}, \"client\"=>nil, \"data\"=>{\"id\"=>\"Q31Q9L1D1FUCKQ\", \"type\"=>\"incident\", \"self\"=>\"https://api.eu.pagerduty.com/incidents/Q31Q9L1D1FUCKQ\", \"html_url\"=>\"https://gi-cloud.eu.pagerduty.com/incidents/Q31Q9L1D1FUCKQ\", \"number\"=>661, \"status\"=>\"acknowledged\", \"incident_key\"=>\"[FILTERED]\", \"created_at\"=>\"2022-11-16T17:39:00Z\", \"title\"=>\"[FILTERED]\", \"service\"=>{\"html_url\"=>\"https://gi-cloud.eu.pagerduty.com/services/P6VSFXU\", \"id\"=>\"P6VSFXU\", \"self\"=>\"https://api.eu.pagerduty.com/services/P6VSFXU\", \"summary\"=>\"GIPCDP - Customer Services\", \"type\"=>\"service_reference\"}, \"assignees\"=>[{\"html_url\"=>\"https://gi-cloud.eu.pagerduty.com/users/PSNIFEK\", \"id\"=>\"PSNIFEK\", \"self\"=>\"https://api.eu.pagerduty.com/users/PSNIFEK\", \"summary\"=>\"Massimiliano Gualtieri\", \"type\"=>\"user_reference\"}], \"escalation_policy\"=>{\"html_url\"=>\"https://gi-cloud.eu.pagerduty.com/escalation_policies/PT0YFSR\", \"id\"=>\"PT0YFSR\", \"self\"=>\"https://api.eu.pagerduty.com/escalation_policies/PT0YFSR\", \"summary\"=>\"Standard Incident Escalation Policy\", \"type\"=>\"escalation_policy_reference\"}, \"teams\"=>[{\"html_url\"=>\"https://gi-cloud.eu.pagerduty.com/teams/PG0PF4W\", \"id\"=>\"PG0PF4W\", \"self\"=>\"https://api.eu.pagerduty.com/teams/PG0PF4W\", \"summary\"=>\"GIPCDP - Team\", \"type\"=>\"team_reference\"}], \"priority\"=>nil, \"urgency\"=>\"high\", \"conference_bridge\"=>nil, \"resolve_reason\"=>nil}}}"
]
}
}Unfortunately, I haven't been able to find any further processing of this type of requests.
Output of checks
This bug happens on GitLab.com
Results of GitLab environment info
Expand for output related to GitLab environment info
(For installations with omnibus-gitlab package run and paste the output of: `sudo gitlab-rake gitlab:env:info`) (For installations from source run and paste the output of: `sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production`)
Results of GitLab application Check
Expand for output related to the GitLab application check
(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:check SANITIZE=true)(For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true)(we will only investigate if the tests are passing)