Guidance for typing large numbers in streams' JSON Schemas
We have run into a few cases recently where very large integers or very large currency amounts were being stored as strings in the source API, and the question has come up in a few forms:
Should we type this to an int or numeric type?
How can we avoid overflow downstream if the contents are not expected to fit into a 32-bit structure?
Example from tap-github
https://github.com/MeltanoLabs/tap-github/pull/93#issuecomment-1073363917
@Ry-DS It looks like the API is indeed returning them as strings, but they are clearly integers... Can we force the conversion? Do you get an error?
-> https://api.github.com/repos/facebook/react/events
It makes sense that it's a string as the number is quite large (20 billion) which definitely won't fit in a 32bit signed integer (max 2 billion). I think it's the right move to keep it as strings (it would error out otherwise).