Pipeline: Saving to BigQuery adds `created_at` field, which breaks schemas
Problem to solve
Proposal
In an effort to do daily runs of duo-chat eval
we added a created_at
field so that we could group the resultant data by day/week/etc. This field was added in a module that is common to all pipelines for writing to BQ. The problem arises when we ran the code-suggestion pipeline which does not have a created_at
timestamp in the schema so the data doesn't save to BigQuery this stranding the data and crashing the run at the end.
The created_at
field setting should be moved into the duo-chat pipeline and not the common BQ writing wrapper.
Links / references
Edited by Stephan Rayner