Project API getting avatars as binary

This is a follow-up of gitlab-ce#18219.

Using curl to download a private project's avatar does not work for me – neither with my actual API token nor with the $CI_JOB_TOKEN.

I would like to download a project's avatar from with in a pipeline to add it to a documentation bundle that will be copied to and published on a different server.