Discussion: Allow start_date to override the state timestamp when it is more recent

I'd like to be able to use the state between runs to minimise how much data we pull back from Gitlab.

Unfortunately the heaviest part for us is tracking back through to the first created status job as this is non terminal. Ultimately this will nearly always track back to one of the first few pipelines for some of our projects because we have manual jobs that are blocking other jobs that are then left in the created status.

To get around this I'm dynamically setting the start_date by running the following command first:

if [ $INCREMENTAL = true ]; then
  export GITLAB_API_START_DATE=$(date --date "1 month ago" --iso-8601=seconds)
else
  export GITLAB_API_START_DATE=2020-01-01T00:00:00Z
fi

This then means we only track back a month on incremental runs when there is no state. However this doesn't work when there is state because it will use the state instead and only falls back to start_date when the state is empty for the entity.

I'd like to propose the following change:

diff --git a/tap_gitlab/__init__.py b/tap_gitlab/__init__.py
index b38a292..a3ccc14 100644
--- a/tap_gitlab/__init__.py
+++ b/tap_gitlab/__init__.py
@@ -169,7 +169,7 @@ def get_url(entity, id, secondary_id=None, start_date=None):
 
 
 def get_start(entity):
-    if entity not in STATE:
+    if entity not in STATE or parse_datetime(STATE[entity]) < parse_datetime(CONFIG['start_date']):
         STATE[entity] = CONFIG['start_date']
     return STATE[entity]

This feels potentially a little bit controversial to me for some reason but if someone states a start_date that is more recent than the state is tracking it is likely because they are deliberately trying to override the state being too old for an entity. I might be missing a use case here though.