cmd/go: go mod tidy, vendor hide module load errors

[rsc]

The module loader leaves some problems for the standard loader to find, like not being able to resolve imports. But 'go mod tidy' and 'go mod vendor' do not run the standard loader, so those problems go undiagnosed. Maybe we should look for them explicitly.

For example make a file x.go that says import "nonexist". go mod tidy succeeds quietly.

[kevinburke]

#27868 is possibly a duplicate of this issue; I'm not positive.

[jsternberg]

We have run into this issue where using go mod tidy on a machine that doesn't have bzr, but that has bzr dependencies, will just not mention that it didn't work correctly.

We caught it because our CI system is now running go mod tidy and it has bzr installed.

[gopherbot]

Change https://golang.org/cl/153459 mentions this issue: cmd/go: fix 'go test' and 'go fmt' with files outside a module

[myitcv]

This might be another instance of this, I'm not 100% sure:

The following:

cd $(mktemp -d)
go mod init mod
cat <<EOD > main.go
package x

import (
        _ "github.com/hashicorp/vault/api"
)

func main() {
}
EOD
go mod tidy
go mod graph | grep labix

gives no results.

But if you then do:

go list all
go mod tidy
go mod graph | grep labix

you get:

mod labix.org/v2/mgo@v0.0.0-20140701140051-000000000287

which I think is as expected because:

$ go mod why -m labix.org/v2/mgo
# labix.org/v2/mgo
mod
github.com/hashicorp/vault/api
github.com/hashicorp/vault/api.test
github.com/hashicorp/vault/helper/builtinplugins
github.com/hashicorp/vault/builtin/logical/nomad
github.com/hashicorp/nomad/api
github.com/hashicorp/nomad/nomad/structs
github.com/hashicorp/go-msgpack/codec
github.com/hashicorp/go-msgpack/codec.test
labix.org/v2/mgo/bson

[leitzler]

@bcmills this issue is tagged with NeedsInvestigation, are there any outstanding questions that need answers before solving this one?

As context I think this is a quite important issue to solve before go1.13 as it adds a lot of confusion for users.

I've seen several cases the thing that happens when bzr isn't installed (as described above by Paul and Jonathan), as well as when indirect dependencies have dependencies that gets removed/broken.

[thepudds]

Hi @leitzler or @jsternberg, I think launchpad.net/gocheck and labix.org/v2/mgo might be examples, but do have a simple set of "from scratch" steps that are an example this with bzr where go mod tidy does not complain? Could be helpful to make sure the ultimate fix here covers that case as well.

[leitzler]

Sure! So far I have three different cases that I think is directly caused by this issue.

1. A Bazaar dependency when the bazaar client bzr isn't installed.

$ docker run -it --rm golang sh
# go version
go version go1.12.4 linux/amd64
# cd $(mktemp -d)
# rm $(which bzr)
# go mod init x
go: creating new go.mod: module x
# cat > main.go
package x

import _ "launchpad.net/gocheck"
# go mod tidy
# cat go.mod
module x

go 1.12
# go build
main.go:3:8: unknown import path "launchpad.net/gocheck": bzr branch --use-existing-dir https://launchpad.net/~niemeyer/gocheck/trunk . in /go/pkg/mod/cache/vcs/f46ce2ae80d31f9b0a29099baa203e3b6d269dace4e5357a2cf74bd109e13339: exec: "bzr": executable file not found in $PATH

2. An direct (or indirect) import can't be found:

$ cd $(mktemp -d) && go mod init x
go: creating new go.mod: module x
$ cat > main.go
package x

import _ "github.com/modsdemo/brokenimport"
$ go mod tidy
go: finding github.com/modsdemo/brokenimport latest
$ cat go.mod
module x

go 1.12

require github.com/modsdemo/brokenimport v0.0.0-20190506193535-03bc7b9ad6c8
$ go build
/home/leitzler/go/pkg/mod/github.com/modsdemo/brokenimport@v0.0.0-20190506193535-03bc7b9ad6c8/brokenimport.go:3:8: unknown import path "github.com/modsdemo/havebeenrenamed": cannot find module providing package github.com/modsdemo/havebeenrenamed           

3. Using invalid GOPROXY setting

$ go mod init x
go: creating new go.mod: module x
$ cat > main.go
package x

import _ "github.com/modsdemo/notags"
$ GOPROXY=proxy.golang.org go mod tidy           # Missing https://, gives no error msg
$ cat go.mod
module x

go 1.12
$ GOPROXY=proxy.golang.org go build              # Missing https://
main.go:3:8: unknown import path "github.com/modsdemo/notags": cannot find module providing package github.com/modsdemo/notags                                  
$ go mod tidy                                    # Works as expected 
go: finding github.com/modsdemo/notags latest
$ go build                                       # Works as expected
$

[thepudds]

@leitzler That's great! Thank you for taking the time to do that.

[bcmills]

@leitzler

are there any outstanding questions that need answers before solving this one?

We need to figure out which paths are missing the needed error checks, and how to add them without breaking things like go list -e.

[rogpeppe]

FWIW I just ran across what I think was this issue when using go mod vendor and it took me ages to work out what the problem was. There was an "ambiguous import" issue in one module; running go mod vendor appeared to succeed but actually didn't vendor the modules that there was a problem with, so we just saw an error about a missing module when CI ran.

[gopherbot]

Change https://golang.org/cl/188763 mentions this issue: cmd/go: report loading errors from 'go mod tidy' and 'go mod vendor'

[bcmills]

The fix for this turned out to be super simple, given the nice import-stack reporting that @jayconrod added earlier in the 1.13 cycle. I'm hoping we can get this into the next 1.13 build.