Skip to content

Serialize JIRA response to avoid field explosion in logs

Igor requested to merge jira-sync-log-cardinality into master

What does this MR do?

As part of https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/10094, @mwasilewski-gitlab and I are looking at the ES logging cluster's cluster state which can grow when there is an explosion in fields.

The JiraConnect::SyncService is logging a ruby hash, which gets serialized to a JSON object. This will produce fields in elasticsearch for each key. If the keys are dynamic, we get an explosion in the number of fields.

You can also see that in the flamegraph from https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/10134:

cluster_state.svg

That flamegraph reveals a lot of entries under:

pubsub-rails-inf-gprd-$suffix.mappings._doc.properties.json.properties.response.properties.acceptedDevinfoEntities.properties

Googling for acceptedDevinfoEntities reveals that this is part of a jira API.

Looking at a sample log line in kibana with that field set, we see:

{"severity":"INFO","time":"2020-05-12T07:48:17.774Z","correlation_id":"m6eGdV7Yst1","integration":"JiraConnect","project_id":[redacted],"project_path":"[redacted]","response":{"acceptedDevinfoEntities":{"[redacted]":{"branches":["[redacted]"],"commits":["[redacted]"],"pullRequests":[]}},"failedDevinfoEntities":{},"unknownIssueKeys":[]},"tag":"rails.integrations","environment":"gprd","hostname":"sidekiq-catchall-06-sv-gprd","fqdn":"sidekiq-catchall-06-sv-gprd.c.gitlab-production.internal","stage":"main","shard":"catchall","tier":"sv","type":"sidekiq","path":null,"message":null,"params":null}

By serializing the value of response to a string, we can avoid each of the acceptedDevinfoEntities keys (which appears to be an id) to become its own field on the elastic side.

This patch also renames response to jira_response to avoid conflicts with other (nested) fields named response.

Screenshots

Does this MR meet the acceptance criteria?

Conformity

Availability and Testing

Security

If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:

  • Label as security and @ mention @gitlab-com/gl-security/appsec
  • The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
  • Security reports checked/validated by a reviewer from the AppSec team
Edited by Igor

Merge request reports