Epoch values should be considered valid timestamps
Summary
From what I can tell, the SDK cannot currently parse epoch-encoded timestamps and instead pushes them through the handle_invalid_timestamp
logic. This is because the timestamp parsing is done using the dateutil.parser.parse
functionality. Example below:
from dateutil import parser
ts = 1634606164.000200
parser.parse(ts)
>>> ERROR: dateutil.parser._parser.ParserError: year 1634606164 is out of range: 1634606164.000200
However, if we use datetime.datetime.fromtimestamp
, the value is parsed successfully.
from datetime import datetime
datetime.fromtimestamp(ts)
>>> datetime.datetime(2021, 10, 18, 21, 16, 4, 200)
In this scenario, I would like to be able to use ts
as a date-time formatted replication key column, but the strategy for doing so is not entirely clear to me.
Proposed benefits
Enabling epoch parsing would allow more timestamp data to be treated in a "standard" way.
Proposal details
I would like to see the SDK handle this case easily, to reduce friction when developing against this type of data.
Best reasons not to build
It is not entirely clear to me whether handling the data as a timestamp is the best way to approach this problem. Should we add a new "format" -- date_epoch
? The major issue I could see is that, if an epoch timestamp is used as the replication key (e.g.), it would need to remain unconverted when being pushed to the external system. (The SDK shouldn't convert epoch -> date_iso8601, and then send that timestamp in the request.)