Private token authentication doesn't work with git-ref artifacts URL

Summary

Usually it is possible to authenticate using the Private token from Profile > Account. This does not work for the git-ref artifacts URL added with #4255 (closed).

Steps to reproduce

Take for example this artifact (from a random project with internal visibility):
https://gitlab.com/afelio-design/ggolfb-contrappception/builds/4726658/artifacts/download

The artifact can also be obtained using this URL:
https://gitlab.com/afelio-design/ggolfb-contrappception/builds/artifacts/master/download?job=pages
(Currently you will get the exact same file, later - when someone pushed to master - you will get a newer version of that file.)

Now if you try this in an incognito window, naturally you will get the sign-in page.

Now append private_token=yourprivatetoken (with yourprivatetoken being your own private token) to get this URL:
https://gitlab.com/afelio-design/ggolfb-contrappception/builds/4726658/artifacts/download?private_token=yourprivatetoken
The download will work.

If you do the same with the git-ref "latest" URL:
https://gitlab.com/afelio-design/ggolfb-contrappception/builds/artifacts/master/download?job=pages&private_token=yourprivatetoken
The download will not work, you'll get the sign-in page.

I would expect the download to work in both cases.