DeepMind released a Experience Replay framework, Reverb.
https://deepmind.com/research/open-source/Reverb