Skip to content

Add check to allow only events which are less than 24 hours old in devkit

Ankit Panchal requested to merge ankit.panchal-main-patch-a377 into main

What does this MR do and why?

This MR updates custom enricher to discard events which are 24 hours old.

custom enricher MR - custom-enrichers!4 (merged) Relates to - gitlab-org/gitlab#434471

How to set up and validate locally

  1. To test if changes are working fine locally, instead of 24 hours change the difference to 10s. Below is the js code that does that.
function process(event) {
  if (!event.getDvce_sent_tstamp()) {
    throw "dvce_sent_tstamp is not defined";
  }

  if (!event.getDvce_created_tstamp()) {
    throw "dvce_created_tstamp is not defined";
  }

  const dvceSentTstamp = new Date(event.getDvce_sent_tstamp());
  const dvceCreatedTstamp = new Date(event.getDvce_created_tstamp());
  const timeDifference = dvceSentTstamp.getTime() - dvceCreatedTstamp.getTime();
  const hoursDifference = timeDifference / (1000 * 60 * 60);

  if (hoursDifference > 0.002) {
    throw `The time difference between dvce_sent_tstamp (${dvceSentTstamp.toISOString()}) and dvce_created_tstamp (${dvceCreatedTstamp.toISOString()}) is greater than 24 hours`;
  }
}

Base64encoded version -

ZnVuY3Rpb24gcHJvY2VzcyhldmVudCkgewogIGlmICghZXZlbnQuZ2V0RHZjZV9zZW50X3RzdGFtcCgpKSB7CiAgICB0aHJvdyAiZHZjZV9zZW50X3RzdGFtcCBpcyBub3QgZGVmaW5lZCI7CiAgfQoKICBpZiAoIWV2ZW50LmdldER2Y2VfY3JlYXRlZF90c3RhbXAoKSkgewogICAgdGhyb3cgImR2Y2VfY3JlYXRlZF90c3RhbXAgaXMgbm90IGRlZmluZWQiOwogIH0KCiAgY29uc3QgZHZjZVNlbnRUc3RhbXAgPSBuZXcgRGF0ZShldmVudC5nZXREdmNlX3NlbnRfdHN0YW1wKCkpOwogIGNvbnN0IGR2Y2VDcmVhdGVkVHN0YW1wID0gbmV3IERhdGUoZXZlbnQuZ2V0RHZjZV9jcmVhdGVkX3RzdGFtcCgpKTsKICBjb25zdCB0aW1lRGlmZmVyZW5jZSA9IGR2Y2VTZW50VHN0YW1wLmdldFRpbWUoKSAtIGR2Y2VDcmVhdGVkVHN0YW1wLmdldFRpbWUoKTsKICBjb25zdCBob3Vyc0RpZmZlcmVuY2UgPSB0aW1lRGlmZmVyZW5jZSAvICgxMDAwICogNjAgKiA2MCk7CgogIGlmIChob3Vyc0RpZmZlcmVuY2UgPiAwLjAwMikgewogICAgdGhyb3cgYFRoZSB0aW1lIGRpZmZlcmVuY2UgYmV0d2VlbiBkdmNlX3NlbnRfdHN0YW1wICgke2R2Y2VTZW50VHN0YW1wLnRvSVNPU3RyaW5nKCl9KSBhbmQgZHZjZV9jcmVhdGVkX3RzdGFtcCAoJHtkdmNlQ3JlYXRlZFRzdGFtcC50b0lTT1N0cmluZygpfSkgaXMgZ3JlYXRlciB0aGFuIDI0IGhvdXJzYDsKICB9Cn0=
  1. Copy base64 encoded version and put it instead of this MR's change.
  2. Send events.
  3. Change dtm field manually so that it is more than 10s than current time. Screenshot_2024-02-22_at_1.46.10_PM
  4. Verify that these events went to snowplow_bad_events table with error message. Screenshot_2024-02-22_at_1.47.53_PM Screenshot_2024-02-22_at_1.47.33_PM

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Ankit Panchal

Merge request reports