I was testing Athens with docker in my local, trying to setup the authentication so that Athens is able to access to company's private module. I followed the guide here and the command go get github.com/SomeCompany/someprivatemodule successfully pulled the latest version of that lib to my local. But there's something weird.
- Those config files (.gitconfig, .ssh/config and .ssh/id_rsa) didn't make that command work. Instead, the .netrc file, consisting of my GitHub token, makes it work. I've tried to run the docker image without mounting those config files and the command still works; However, if I removed ~/.netrc in my local, the
go getcommand fails. It seems go get is still depending on local config instead of Athens.
go get command fails when .netrc removed
- The
go getcommand does pull the private module from GitHub. However, under ATHENS_DISK_STORAGE_ROOT, I can only find those public modules that github.com/SomeCompany/someprivatemodule depends on, and those private modules can only be found under GOPATH/pkg/mod. Isn't Athens supposed to download the private module to ATHENS_DISK_STORAGE_ROOT, so that once GitHub/gitlab is down, users can still download those private modules from Athens?
Missing private module in Athens dir
Here is my setup:
GOPROXY=127.0.0.1:3000, which is the exposed port of Athens. GOPRIVATE="github.com/SomeCompany"
GONOPROXY="github.com/SomeCompany"
GONOSUMDB="github.com/SomeCompany"
~/.netrc
machine github.com login some_personal_access_token
.gitconfig
[url "ssh://[email protected]/"]
insteadOf = https://github.com/
[url "ssh://[email protected]/"]
insteadOf = https://gitlab.com/
.ssh/config
Host github.com
Hostname github.com
StrictHostKeyChecking no
IdentityFile /root/.ssh/id_rsa
command of running docker image
docker run -d -v $ATHENS_STORAGE:/var/lib/athens -e ATHENS_DISK_STORAGE_ROOT=/var/lib/athens -e ATHENS_STORAGE_TYPE=disk -v $PWD/.gitconfig:/root/.gitconfig -v @PWD/.ssh:/root/.ssh --name athens-proxy --restart always -p 3000:3000 gomods/athens:v0.11.0
1st edit
By setting GOPRIVATE to empty, GONOPROXY and GONOSUMDB get unset automatically. And I tried to rerun the docker with original setup, I got the 404 error:
Docker log:
INFO[2:35PM]: Exporter not specified. Traces won't be exported
2021-05-29 14:35:11.957339 I | Starting application at port :3000
INFO[2:35PM]: exit status 1: go list -m: github.com/SomeCompany@latest: invalid github.com/ import path "github.com/SomeCompany"
http-method=GET http-path=/github.com/SomeCompany/@v/list kind=Not Found module= operation=download.ListHandler ops=[download.ListHandler pool.List protocol.List vcsLister.List] request-id=3660faa3-256f-43fb-9937-f29565e7afa9 version=
INFO[2:35PM]: incoming request http-method=GET http-path=/github.com/SomeCompany/@v/list http-status=404 request-id=3660faa3-256f-43fb-9937-f29565e7afa9
INFO[2:35PM]: exit status 1: go list -m: github.com@latest: unrecognized import path "github.com": parse https://github.com/?go-get=1: no go-import meta tags ()
http-method=GET http-path=/github.com/@v/list kind=Not Found module= operation=download.ListHandler ops=[download.ListHandler pool.List protocol.List vcsLister.List] request-id=4d6659d2-710d-445f-b07f-7407aa4f2e3e version=
INFO[2:35PM]: incoming request http-method=GET http-path=/github.com/@v/list http-status=404 request-id=4d6659d2-710d-445f-b07f-7407aa4f2e3e
INFO[2:35PM]: incoming request http-method=GET http-path=/github.com/SomeCompany/someprivatemodule/@v/list http-status=200 request-id=e8d95d77-81dc-46fe-9523-e35f6dc8201f
DEBUG[2:35PM]: saving github.com/SomeCompany/[email protected] to storage... http-method=GET http-path=/github.com/SomeCompany/someprivatemodule/@v/v1.0.5.info request-id=24964b9c-b0b2-450a-bd9a-9d59f60a5c52
INFO[2:35PM]: github.com/SomeCompany/[email protected]/go.mod: verifying module: github.com/SomeCompany/[email protected]/go.mod: reading https://sum.golang.org/lookup/github.com/SomeCompany/[email protected]: 410 Gone
server response: not found: github.com/SomeCompany/[email protected]: invalid version: unknown revision v1.0.5 http-method=GET http-path=/github.com/SomeCompany/someprivatemodule/@v/v1.0.5.info kind=Not Found module=github.com/SomeCompany/someprivatemodule operation=download.InfoHandler ops=[download.InfoHandler pool.Info protocol.Info protocol.processDownload stash.Pool stasher.Stash stasher.fetchModule goGetFetcher.Fetch module.downloadModule] request-id=24964b9c-b0b2-450a-bd9a-9d59f60a5c52 version=v1.0.5
INFO[2:35PM]: incoming request http-method=GET http-path=/github.com/SomeCompany/someprivatemodule/@v/v1.0.5.info http-status=404 request-id=24964b9c-b0b2-450a-bd9a-9d59f60a5c52
It seems Athens tries to find packages from sum.golang.org. So I tried to mount .netrc file with proper credentials and set ATHENS_NETRC_PATH. This still gives me same error.
Another fact: with v0.11.0, I couldn't download anything with go get. However, with v0.2.0 version of Athens docker image, I was able to download part of the modules. The docker log (v0.2.0) is shown below:
buffalo: Unless you set SESSION_SECRET env variable, your session storage is not protected!
time="2021-05-29T19:39:20Z" level=info msg="Exporter not specified. Traces won't be exported"
buffalo: Starting application at :3000
time="2021-05-29T19:39:26Z" level=error msg="exit status 1: go list -m github.com/SomeCompany: invalid github.com/ import path \"github.com/SomeCompany\"\n" http-method=GET http-path="/github.com/SomeCompany/@v/list/" http-url="/github.com/SomeCompany/@v/list/" kind="Internal Server Error" module= operation=download.ListHandler ops="[download.ListHandler pool.List protocol.List vcsLister.List]" version=
handler: GET /github.com/SomeCompany/@v/list/ [500]
time="2021-05-29T19:39:26Z" level=error msg="exit status 1: go list -m github.com: unrecognized import path \"github.com\" (parse https://github.com?go-get=1: no go-import meta tags ())\n" http-method=GET http-path=/github.com/@v/list/ http-url=/github.com/@v/list/ kind="Internal Server Error" module= operation=download.ListHandler ops="[download.ListHandler pool.List protocol.List vcsLister.List]" version=
handler: GET /github.com/@v/list/ [500]
handler: GET /github.com/SomeCompany/someprivatemodule/@v/list/ [200]
time="2021-05-29T19:39:26Z" level=error msg="exit status 1: go: finding github.com/SomeCompany/someprivatemodule/somefile latest\ngo list -m github.com/SomeCompany/someprivatemodule/somefile: no matching versions for query \"latest\"\n" http-method=GET http-path="/github.com/SomeCompany/someprivatemodule/somefile/@v/list/" http-url="/github.com/SomeCompany/someprivatemodule/somefile/@v/list/" kind="Internal Server Error" module= operation=download.ListHandler ops="[download.ListHandler pool.List protocol.List vcsLister.List]" version=
handler: GET /github.com/SomeCompany/someprivatemodule/ratelimit/@v/list/ [500]
handler: GET /github.com/SomeCompany/someprivatemodule/@v/v1.0.5.info [200]
handler: GET /github.com/SomeCompany/someprivatemodule/@v/v1.0.5.mod [200]
handler: GET /github.com/SomeCompany/someprivatemodule/@v/v1.0.5.zip [200]
And apparently this still won't work cuz its missing part of the module.
Edit: Solved by adding GONOSUM_PATTERNS
I just tried Athens a bit. It serves as a Go module cache. When you start athens and set GOPROXY=127.0.0.1:3000, the go get command would try to get modules using the sequence below:
Here your GOPROXY is athens server, thus your local go get will send request to athens. Then athens will check if the module is in ATHENS_DISK_STORAGE_ROOT; if not, get the proxy from github.com or wherever it should be.
You can try to walkthrough the process.
As for your questions,
I notice that you set
github.com/SomeCompanyin your GONOPROXY, thus all modules in this repository will not use GOPROXY, and athens will not work as well.That's why in Question1, your .netrc is used; and in Question2, only the modules that
github.com/SomeCompany/someprivatemoduledepends on is downloaded from athens, but the module itself is not.Remove the GONOPROXY and try again, see what's happening.