Allow go-get meta information (for use with go's `dep init`) to be correct when using personal-token (using .netrc) and with a gitlab-setup with subprojects
Description
Hi all!
go-get has a long story with issues for private repositories, especially when using subprojects in gitlab. Whenever I talk about a repo from now on, it is meant as a gitlab repo which is private using a structure with subprojects. Currently the only option to make it work is to append .git in the manifest. Why this is still problematic is written later. Also to point out, this previous linked answer compiles the reasoning behind it and I totally agree with it, in short:
- unauthenticated users shouldn't see repo structure
- when authenticated, users can see repo structure they've access to and thus the go-get meta information is correct also for private repos (except see shortly below)
So far so good. I don't know if this is a bug or not (thus currently a feature proposal), but go-get meta information is still wrong when using personal token with a .netrc file:
Expected result
(as taken with my browser, authenticated on my gitlab): view-source:https://mygitlab.io/group/subgroup/repo?go-get=1
<html><head><meta name="go-import" content="mygitlab.io/group/subgroup/repo git https://mygitlab.io/group/subgroup/repo.git" /></head></html>
Current result (authenticated)
Using a .netrc file allows you to clone a gitlab repo with your personal-token, but you still get the wrong go-get meta information (-n in curl is for using the .netrc file):
$ cat ~/.netrc
machine mygitlab.io
login myuser
password correct-personal-token
$ git clone https://mygitlab.io/group/subgroup/repo.git
Cloning into 'repo'...
remote: Counting objects: 370, done.
remote: Compressing objects: 100% (212/212), done.
remote: Total 370 (delta 148), reused 343 (delta 124)
Receiving objects: 100% (370/370), 61.90 KiB | 0 bytes/s, done.
Resolving deltas: 100% (148/148), done.
Checking connectivity... done.
$ go get -v mygitlab.io/group/subgroup/repo.git # here with .git as to show that also go get can work with .netrc, but the .git suffix is problematic
mygitlab.io/group/subgroup/repo.git (download)
mygitlab.io/group/subgroup/repo.git
$ curl -n https://mygitlab.io/group/subgroup/repo?go-get=1
<html><head><meta name="go-import" content="mygitlab.io/group/subgroup git https://mygitlab.io/group/subgroup.git" /></head></html>
Current result (unauthenticated)
And to "prove" that this is the only thing manipulating the way (no .gitconfig etc). Also now i'ts correct that the go-get meta information are wrong, as I'm unauthenticated:
$ cat ~/.netrc
machine mygitlab.io
login myuser
password bogus-personal-token
$ git clone https://mygitlab.io/group/subgroup/repo.git
Cloning into 'repo'...
Username for 'https://mygitlab.io': ^C
$ go get -v mygitlab.io/group/subgroup/repo.git # the .git suffix itself is still problematic, just for completeness how it reacts to bogus .netrc
Username for 'https://mygitlab.io': ^C
$ curl -n https://mygitlab.io/group/subgroup/repo?go-get=1
<html><head><meta name="go-import" content="mygitlab.io/group/subgroup git https://mygitlab.io/group/subgroup.git" /></head></html>
A little conclusion so far: Using the .netrc file allows me to pull a repo from gitlab (and thus knowing that this repo path is correct), but using the .netrc file still doesn't give me the correct go-get meta information.
And don't point the finger to the .netrc file ;) The header PRIVATE-TOKEN is also not working:
$ curl --header "PRIVATE-TOKEN: correct-personal-token" https://mygitlab.io/group/subgroup/repo?go-get=1
<html><head><meta name="go-import" content="mygitlab.io/group/subgroup git https://mygitlab.io/group/subgroup.git" /></head></html>
Current workarounds not sufficient
Why is the .git workaround not sufficient you might ask?
Intro:
Basically everything is dependent on go get
, when this works flawlessly given correct authentication, everything else should also work. Below the dep init/ensure
part is additional context, but the focus should be on getting go get
to work. dep
is only in one thing better: differntiate the name of the package with the source uri, but this has still some flaws.
The problem is simple:
Using go get
with a path suffixing .git assumes that go package has that package named repo.git, but it should be without the .git suffix. So a no-go.
The problem is complex:
We have some control when using a dependency manager, golangs official dep for now. I have some big go source where I want to pull the dependencies from scratch, with a lot of them sitting behind such kind of gitlab. I would have to do "a lot" of manual work to write the dep .toml file. Since I'm initializing the dependencies, I have to use dep init
, but it doesn't do anything because of "broken" packages. The partial best-effort output is still not implemented, otherwise I could add them later using the more convenient dep ensure -add ...
command with the --source
argument. I can't use dep ensure -add ...
now, as i have to do a dep init
first.
$ dep ensure -add mygitlab.io/group/subgroup/repo
could not find project Gopkg.toml, use dep init to initiate a manifest
Why this proposal would solve all problems
Now there's the catch: go dep support's the .netrc file, so if only gitlab would also return the correct go-get meta information, everything should work flawlessly as each tooling now is using that .netrc file to authenticate the user to get it's information as authenticated user, whatever it is.
Proposal
Support .netrc file to return the correct go-get meta information, as the user is authenticated. optionally also look into the --header "PRIVATE-TOKEN" as this also won't return them correctly. Also, I assume the new CI permissions model on dependent repositories also will have this flaw with go projects when not addressed (as it's using the .netrc file).
Or allow the administrator to mark a flag, that it allows to return the correct go-get information even when unauthorized. B you'll need to make a warning box then, as it is somewhat possible to get information about the repo tree.
Links / references
All directly in the text inline