Serialize JIRA response to avoid field explosion in logs
What does this MR do?
As part of https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/10094, @mwasilewski-gitlab and I are looking at the ES logging cluster's cluster state which can grow when there is an explosion in fields.
The JiraConnect::SyncService
is logging a ruby hash, which gets serialized to a JSON object. This will produce fields in elasticsearch for each key. If the keys are dynamic, we get an explosion in the number of fields.
You can also see that in the flamegraph from https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/10134:
That flamegraph reveals a lot of entries under:
pubsub-rails-inf-gprd-$suffix.mappings._doc.properties.json.properties.response.properties.acceptedDevinfoEntities.properties
Googling for acceptedDevinfoEntities
reveals that this is part of a jira API.
Looking at a sample log line in kibana with that field set, we see:
{"severity":"INFO","time":"2020-05-12T07:48:17.774Z","correlation_id":"m6eGdV7Yst1","integration":"JiraConnect","project_id":[redacted],"project_path":"[redacted]","response":{"acceptedDevinfoEntities":{"[redacted]":{"branches":["[redacted]"],"commits":["[redacted]"],"pullRequests":[]}},"failedDevinfoEntities":{},"unknownIssueKeys":[]},"tag":"rails.integrations","environment":"gprd","hostname":"sidekiq-catchall-06-sv-gprd","fqdn":"sidekiq-catchall-06-sv-gprd.c.gitlab-production.internal","stage":"main","shard":"catchall","tier":"sv","type":"sidekiq","path":null,"message":null,"params":null}
By serializing the value of response
to a string, we can avoid each of the acceptedDevinfoEntities
keys (which appears to be an id) to become its own field on the elastic side.
This patch also renames response
to jira_response
to avoid conflicts with other (nested) fields named response
.
Screenshots
Does this MR meet the acceptance criteria?
Conformity
-
Changelog entry -
Documentation (if required) -
Code review guidelines -
Merge request performance guidelines -
Style guides -
Database guides -
Separation of EE specific content
Availability and Testing
-
Review and add/update tests for this feature/bug. Consider all test levels. See the Test Planning Process. -
Tested in all supported browsers -
Informed Infrastructure department of a default or new setting change, if applicable per definition of done
Security
If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:
-
Label as security and @ mention @gitlab-com/gl-security/appsec
-
The MR includes necessary changes to maintain consistency between UI, API, email, or other methods -
Security reports checked/validated by a reviewer from the AppSec team