Use materialized view over Geo FDW tables
This idea was raised by @_stark on chat:
I thought of another idea which might be helpful. If you use materialized views over the fdw tables you could refresh them on a schedule of your choosing, have special indexes on them to aid the geo queries, and the plans might be equivalent to what you would get with aggregate pushdown since it would all be running on the tracking database.
Reference: https://www.postgresql.org/docs/9.6/static/rules-materializedviews.html