Platform: Add query API transformation layer
Problem to solve
With Platform: Create Protobuf-over-gRPC receiver (#21 - closed) we will have created a gRPC service. We now need to create a transformation layer which takes the protobuf definition and turns it into the appropriate Clickhouse query. The Clickhouse response should then be formatted and sent back through the gRPC service.
It is worth noting that queries could span from simpler count queries, to far more complex aggregated material views. So we should design the transformation layer to iterate towards supporting complex queries.
Technical considerations
As per the design document, we must consider:
- We will need to eventually support streams for instances when a lot of data is being queried.
- We require all queries to have a sorting field (this can be defaulted in the proto definition or query transformation).
- We require all aggregation queries to have a grouping field (this can be defaulted in the proto definition or query transformation).
- We must use keyset pagination.
- We will be recording gRPC request metrics.
Edited by Robert Hunt