Apache Beam client library
Problem to solve and solution
Apache Beam is very powerful stream processing engine. Advanced users could use it to build very advanced derivations of Beneath streams.
Proposed solution and changes
- Add a Beam source and sink to the Python library
Risks and challenges
- Consider other languages (especially in light of portable runners)
- The dataflow runner previously didn't support custom sources in streaming mode in Python. Is that still the case?
Optional: What does success look like, and how can we measure it?
You should be able to deploy a Beam pipeline to Google Cloud dataflow that derives from one stream on Beneath and outputs to a new, derived stream on Beneath.