Caching directories inside $HOME

As far as I can see, currently cache supports only caching of files inside current directory (or by absolute path). But Gradle places it's caches inside ~/.gradle directory. Currently I had tried:

cache:
  paths:
  - ~/.gradle/caches
  - ~/.gradle/wrapper

and

cache:
  paths:
  - $HOME/.gradle/caches
  - $HOME/.gradle/wrapper

But neither of these works - runner just outputs INFO[0000] No files to archive. and does not cache anything.