Update Snowflake timestamp used in extraction and increase frequency of load
-
usage_billing_enrichedis currently extracted based on the value present in theTimestampcolumn (ie. we export a record ifTimestampis set to a time on the day before the day of extraction). I think it is a good idea to switch toenriched_atfor the incremental extraction inusage_billing_enriched. My understanding is that events are created first and enriched shortly(?) afterward. If the extraction job filters on the event’s creationTimestamp, we could miss events that were created before the extraction ran but only became enriched afterward, since the extractor has already moved past that timestamp window. UsingEnrichedAtshould avoid this issue -
We are currently exporting data from ClickHouse to Snowflake once daily. Given the high visibility of this data, I'd like to explore exporting it every 6 hours instead. This would provide more timely updates and prevent file sizes from becoming too large as data volumes grow