Use gx for iptb #2545

chriscool · 2016-04-11T09:01:06Z

This is to fix issue #2525 by using gx to manage the iptb dependency.

chriscool · 2016-04-11T12:12:24Z

I followed the instructions on the gx README (https://github.com/whyrusleeping/gx/blob/master/README.md) and it works on my machine, but it looks like it fails elsewhere because it cannot find the package QmccSTNEUYSJsDSHZj3uKS3pPYCRXCNxr4X8Ub1TQyKPsZ:

ERROR: from shell.Get():  unexpected EOF

retrying fetch QmccSTNEUYSJsDSHZj3uKS3pPYCRXCNxr4X8Ub1TQyKPsZ after a second...

ipfs-shell: warning! unhandled response encoding: text/htmlERROR: from shell.Get():  get: unknown ipfs-shell error encoding: "text/html" - "<html>\r\n<head><title>502 Bad Gateway</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>502 Bad Gateway</h1></center>\r\n<hr><center>nginx/1.9.3</center>\r\n</body>\r\n</html>\r\n"

chriscool · 2016-04-11T15:28:18Z

I can see something on my machine:

> ipfs ls QmccSTNEUYSJsDSHZj3uKS3pPYCRXCNxr4X8Ub1TQyKPsZ
QmRK98cso22c13zpr92n4QSYtrWeqHQXV1RrxvB2wzt7KA 10822786 iptb/

and on the public gateway too: https://gateway.ipfs.io/ipfs/QmccSTNEUYSJsDSHZj3uKS3pPYCRXCNxr4X8Ub1TQyKPsZ

chriscool · 2016-04-11T18:48:16Z

There are still some problems with getting the package.

whyrusleeping · 2016-04-12T16:37:19Z

@chriscool it looks like the gx package can be fetched properly, now the makefile in test/sharness is having trouble finding where the code is to build it.

chriscool · 2016-04-15T18:56:21Z

@whyrusleeping maybe that's true for the CircleCi build, but for the Travis builds, one of them worked and in another one the error is:

ERROR: from shell.Get():  unexpected EOF
retrying fetch QmccSTNEUYSJsDSHZj3uKS3pPYCRXCNxr4X8Ub1TQyKPsZ after a second...
ipfs-shell: warning! unhandled response encoding: text/htmlERROR: from shell.Get():  get: unknown ipfs-shell error encoding: "text/html" - "<html>\r\n<head><title>502 Bad Gateway</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>502 Bad Gateway</h1></center>\r\n<hr><center>nginx/1.9.3</center>\r\n</body>\r\n</html>\r\n"
retrying fetch QmccSTNEUYSJsDSHZj3uKS3pPYCRXCNxr4X8Ub1TQyKPsZ after a second...
ipfs-shell: warning! unhandled response encoding: text/htmlERROR: from shell.Get():  get: unknown ipfs-shell error encoding: "text/html" - "<html>\r\n<head><title>502 Bad Gateway</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>502 Bad Gateway</h1></center>\r\n<hr><center>nginx/1.9.3</center>\r\n</body>\r\n</html>\r\n"
ERROR: [7 / 7 ] parallel fetch: failed to fetch package: QmccSTNEUYSJsDSHZj3uKS3pPYCRXCNxr4X8Ub1TQyKPsZ: get: unknown ipfs-shell error encoding: "text/html" - "<html>\r\n<head><title>502 Bad Gateway</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>502 Bad Gateway</h1></center>\r\n<hr><center>nginx/1.9.3</center>\r\n</body>\r\n</html>\r\n"
ERROR: install deps: failed to fetch dependencies
make: *** [deps] Error 1

chriscool · 2016-04-15T19:53:20Z

Also yeah it looks like when sharness tests are run alone on a CI machine, we will need to run gx install.

chriscool · 2016-04-20T06:50:37Z

So I added a commit that uses the root Makefile to check that everything is ok and maybe install missing packages using gx if not.

chriscool · 2016-04-20T07:37:33Z

On the CircleCi failure we have:

  - go-ipfs depends on iptb (QmccSTNEUYSJsDSHZj3uKS3pPYCRXCNxr4X8Ub1TQyKPsZ)
installing package: iptb-0.0.0
successfully found all deps for iptb
installation of iptb complete!

so it looks like gx installs iptb correctly.

But then we have:

*** installing bin/iptb ***
go build  -o bin/iptb ../../../../../src/github.com/whyrusleeping/iptb
can't load package: package github.com/whyrusleeping/iptb: cannot find package "github.com/whyrusleeping/iptb" in any of:
    /usr/local/go/src/github.com/whyrusleeping/iptb (from $GOROOT)
    /home/ubuntu/.go_workspace/src/github.com/whyrusleeping/iptb (from $GOPATH)
make[2]: *** [bin/iptb] Error 1
make[1]: *** [bin/iptb] Error 2
make: *** [test_sharness_expensive] Error 2

so we cannot find iptb.

Where was it installed in the first place?

whyrusleeping · 2016-04-20T16:02:29Z

now that we are using gx, the import path of iptb is no longer github.com/whyrusleeping/iptb its the gx path, so we will need to: go build gx/ipfs/QmccSTNEUYSJsDSHZj3uKS3pPYCRXCNxr4X8Ub1TQyKPsZ/iptb

chriscool · 2016-04-23T17:24:15Z

@whyrusleeping is there a command (maybe gx view something) to get "QmccSTNEUYSJsDSHZj3uKS3pPYCRXCNxr4X8Ub1TQyKPsZ" so that the hash is not hardcoded?

whyrusleeping · 2016-04-23T21:30:50Z

@chriscool i think the best way to do that right now is to do something like gx deps | grep iptb | awk '{ print $2 }'

I need to add something to gx deps to make that query easier

License: MIT Signed-off-by: Christian Couder <chriscool@tuxfamily.org>

Now that iptb has been added as a gx dependency, we can replace existing dependencies with the one from gx. License: MIT Signed-off-by: Christian Couder <chriscool@tuxfamily.org>

License: MIT Signed-off-by: Christian Couder <chriscool@tuxfamily.org>

We need to make sure that go, gx, gx-go are properly installed. License: MIT Signed-off-by: Christian Couder <chriscool@tuxfamily.org>

chriscool · 2016-04-30T05:29:04Z

I haven't yet started to use gx deps find, but at least I rebased everything and made things work on my laptop.

chriscool · 2016-04-30T06:02:17Z

@whyrusleeping in e79e1d3, you changed the following line in test/Makfile:

-MULTIHASH_SRC = ../Godeps/_workspace/src/github.com/jbenet/go-multihash
+MULTIHASH_SRC = ../../../../gx/ipfs/QmYf7ng2hG5XBtJA3tN34DQ2GUN5HNksEw1rLDkmr6vGku/go-multihash

but it looks like go-multihash is not in package.json. Did you forgot to add it there or is it normal?

License: MIT Signed-off-by: Christian Couder <chriscool@tuxfamily.org>

chriscool · 2016-04-30T06:56:05Z

Ok, the last 2 commits should hopefully fix the 2 problems I talk about above, that are the multihash not in package.json and using gx deps find for gx dependencies.

chriscool · 2016-05-01T06:32:53Z

There is now this CircleCI error:

*** installing bin/iptb ***
go build  -o bin/iptb ../../../../gx/ipfs/QmccSTNEUYSJsDSHZj3uKS3pPYCRXCNxr4X8Ub1TQyKPsZ/iptb
../../../../gx/ipfs/QmccSTNEUYSJsDSHZj3uKS3pPYCRXCNxr4X8Ub1TQyKPsZ/iptb/util/util.go:22:2: cannot find package "github.com/jbenet/go-multiaddr-net/Godeps/_workspace/src/github.com/jbenet/go-multiaddr" in any of:
    /usr/local/go/src/github.com/jbenet/go-multiaddr-net/Godeps/_workspace/src/github.com/jbenet/go-multiaddr (from $GOROOT)
    /home/ubuntu/.go_workspace/src/github.com/jbenet/go-multiaddr-net/Godeps/_workspace/src/github.com/jbenet/go-multiaddr (from $GOPATH)
make[2]: *** [bin/iptb] Error 1
make[2]: Leaving directory `/home/ubuntu/.go_workspace/src/github.com/ipfs/go-ipfs/test'
make[1]: *** [bin/iptb] Error 2
make[1]: Leaving directory `/home/ubuntu/.go_workspace/src/github.com/ipfs/go-ipfs/test/sharness'
make: *** [test_sharness_expensive] Error 2

chriscool · 2016-05-01T06:34:08Z

TravisCI errors are still "No output has been received in the last 10m0s, this potentially indicates a stalled build or something wrong with the build itself."

chriscool · 2016-05-01T06:35:12Z

Teamcity error is:

[06:51:34][Step 1/3] [10 / 12] fetched dep: go-libp2p
[06:51:34][Step 1/3] [11 / 12] fetched dep: goprocess
[07:21:34][Step 1/3] ERROR: from shell.Get():  get: unknown ipfs-shell error encoding: "text/html" - "<html>\r\n<head><title>504 Gateway Time-out</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>504 Gateway Time-out</h1></center>\r\n<hr><center>nginx/1.9.3</center>\r\n</body>\r\n</html>\r\n"
[07:21:34][Step 1/3] retrying fetch QmccSTNEUYSJsDSHZj3uKS3pPYCRXCNxr4X8Ub1TQyKPsZ after a second...
[07:51:35][Step 1/3] ERROR: from shell.Get():  get: unknown ipfs-shell error encoding: "text/html" - "<html>\r\n<head><title>504 Gateway Time-out</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>504 Gateway Time-out</h1></center>\r\n<hr><center>nginx/1.9.3</center>\r\n</body>\r\n</html>\r\n"
[07:51:35][Step 1/3] retrying fetch QmccSTNEUYSJsDSHZj3uKS3pPYCRXCNxr4X8Ub1TQyKPsZ after a second...
[08:21:40][Step 1/3] ERROR: from shell.Get():  unexpected EOF
[08:21:40][Step 1/3] ERROR: [12 / 12 ] parallel fetch: failed to fetch package: QmccSTNEUYSJsDSHZj3uKS3pPYCRXCNxr4X8Ub1TQyKPsZ: unexpected EOF
[08:21:40][Step 1/3] ERROR: install deps: failed to fetch dependencies
[08:21:40][Step 1/3] Makefile:34: recipe for target 'deps' failed
[08:21:40][Step 1/3] ipfs-shell: warning! unhandled response encoding: text/htmlipfs-shell: warning! unhandled response encoding: text/htmlmake: *** [deps] Error 1
[08:21:40][Step 1/3] Process exited with code 2
[08:21:40][Step 1/3] Step build (Command Line) failed

It looks like a timeout.

chriscool · 2016-05-01T20:04:53Z

@lgierth could you take a look at the timeouts?
@whyrusleeping do you have an idea about how the CircleCI error could be fixed?

whyrusleeping · 2016-05-02T15:11:04Z

@chriscool the timeouts again just look like you created a new package and it took a while for the gateways to get it. Restarting the tests seems to be working as expected

whyrusleeping · 2016-05-02T16:04:28Z

@chriscool try using Qmbi7xdiz4fJRxTjzanYHJhvpCPRK6yyVrgZmV7ZfvnMiV for iptb. Its a latest version without the godeps multiaddr problem

chriscool · 2016-05-02T17:14:23Z

@whyrusleeping thanks I will take a look soon.

whyrusleeping · 2016-05-02T17:17:17Z

@chriscool if you push this as a branch in this repo I could hack on it a bit too

chriscool · 2016-05-02T19:24:27Z

@whyrusleeping I get the following error on my machine when using Qmbi7xdiz4fJRxTjzanYHJhvpCPRK6yyVrgZmV7ZfvnMiV:

*** installing bin/iptb ***
go build  -o bin/iptb ../../../../gx/ipfs/Qmbi7xdiz4fJRxTjzanYHJhvpCPRK6yyVrgZmV7ZfvnMiV/iptb
# gx/ipfs/Qmbi7xdiz4fJRxTjzanYHJhvpCPRK6yyVrgZmV7ZfvnMiV/iptb/util
../../../../gx/ipfs/Qmbi7xdiz4fJRxTjzanYHJhvpCPRK6yyVrgZmV7ZfvnMiV/iptb/util/util.go:408: cannot use maddr (type "github.com/jbenet/go-multiaddr".Multiaddr) as type "github.com/jbenet/go-multiaddr-net/Godeps/_workspace/src/github.com/jbenet/go-multiaddr".Multiaddr in argument to manet.DialArgs:
        "github.com/jbenet/go-multiaddr".Multiaddr does not implement "github.com/jbenet/go-multiaddr-net/Godeps/_workspace/src/github.com/jbenet/go-multiaddr".Multiaddr (wrong type for Decapsulate method)
                have Decapsulate("github.com/jbenet/go-multiaddr".Multiaddr) "github.com/jbenet/go-multiaddr".Multiaddr
                want Decapsulate("github.com/jbenet/go-multiaddr-net/Godeps/_workspace/src/github.com/jbenet/go-multiaddr".Multiaddr) "github.com/jbenet/go-multiaddr-net/Godeps/_workspace/src/github.com/jbenet/go-multiaddr".Multiaddr
../../../../gx/ipfs/Qmbi7xdiz4fJRxTjzanYHJhvpCPRK6yyVrgZmV7ZfvnMiV/iptb/util/util.go:460: cannot use maddr (type "github.com/jbenet/go-multiaddr".Multiaddr) as type "github.com/jbenet/go-multiaddr-net/Godeps/_workspace/src/github.com/jbenet/go-multiaddr".Multiaddr in argument to manet.DialArgs:
        "github.com/jbenet/go-multiaddr".Multiaddr does not implement "github.com/jbenet/go-multiaddr-net/Godeps/_workspace/src/github.com/jbenet/go-multiaddr".Multiaddr (wrong type for Decapsulate method)
                have Decapsulate("github.com/jbenet/go-multiaddr".Multiaddr) "github.com/jbenet/go-multiaddr".Multiaddr
                want Decapsulate("github.com/jbenet/go-multiaddr-net/Godeps/_workspace/src/github.com/jbenet/go-multiaddr".Multiaddr) "github.com/jbenet/go-multiaddr-net/Godeps/_workspace/src/github.com/jbenet/go-multiaddr".Multiaddr
make: *** [bin/iptb] Erreur 2

chriscool · 2016-05-02T19:29:59Z

@whyrusleeping Sorry about using a branch in my repo for this. I pushed the branch into this repo: https://github.com/ipfs/go-ipfs/commits/use-gx-for-iptb but I don't think it's possible to replace the branch in my repo with the branch in this repo in the PR.

chriscool · 2016-05-02T19:30:26Z

Do you want me to open another PR instead?

License: MIT Signed-off-by: Jeromy <why@ipfs.io>

whyrusleeping · 2016-05-02T19:41:13Z

@chriscool thats okay, I think i've got it now: 1770badbb31e

Pull that commit over and give it a shot.

whyrusleeping · 2016-05-02T19:47:24Z

(or just copy the hash over, i realize pulling that commit it might be more effort than just changing one line)

chriscool · 2016-05-02T20:07:05Z

Ok I pulled from the chriscool-use-gx-for-iptb branch and got your commit. It works here indeed, thanks!

chriscool · 2016-05-02T20:27:12Z

The TravisCI build is still full of "No output has been received in the last 10m0s, this potentially indicates a stalled build or something wrong with the build itself." errors.

whyrusleeping · 2016-05-02T20:28:12Z

the gateways keep running into a yamux bug... i'm working on fixing it now but it makes some of them unresponsive until we reboot them.

chriscool · 2016-05-02T20:29:27Z

CircleCI has this interesting error:

expecting success: 
  ipfsi 3 dht query banana >actual &&
  ipfsi 3 dht query apple >>actual &&
  ipfsi 3 dht query pear >>actual &&
  PEERS=$(wc -l actual | cut -d ' ' -f 1) &&
  [ -s actual ] ||
    test_fsh cat actual

> cat actual

not ok 14 - query
#   
#     ipfsi 3 dht query banana >actual &&
#     ipfsi 3 dht query apple >>actual &&
#     ipfsi 3 dht query pear >>actual &&
#     PEERS=$(wc -l actual | cut -d ' ' -f 1) &&
#     [ -s actual ] ||
#       test_fsh cat actual
#   

expecting success: 
  iptb stop

DEPRECATED Action signature.  Must be `cli.ActionFunc`.  This is an error in the application.  Please contact the distributor of this application if this is not you.  See https://github.com/codegangsta/cli/blob/master/CHANGELOG.md#deprecated-cli-app-action-signature

whyrusleeping · 2016-05-02T20:32:18Z

thats: #2620

And the the DEPRECATED thing is an upstream change in codegangsta/cli (the commands library i use for gx and gx-go). Nothing real to worry about there

chriscool · 2016-05-02T20:41:26Z

Ok thanks. By the way Teamcity has this error:

[20:22:12][Step 3/3] fatal: Program gx is not installed!
[20:22:12][Step 3/3] => Upgrade or install gx using your package manager or run `make gx_upgrade`
[20:22:12][Step 3/3] make[2]: *** [gx_check] Error 1
[20:22:12][Step 3/3] make[1]: *** [global-deps] Error 2

chriscool · 2016-05-03T20:54:07Z

Now there is only one TravisCI build failing, great!

The failure is the usual "No output has been received in the last 10m0s, this potentially indicates a stalled build or something wrong with the build itself."

chriscool mentioned this pull request Apr 11, 2016

Use gx for iptb whyrusleeping/todo#16

Closed

chriscool force-pushed the use-gx-for-iptb branch from 81a561e to fe9035f Compare April 11, 2016 17:01

chriscool mentioned this pull request Apr 25, 2016

main: add gx deps find <dep> whyrusleeping/gx#70

Merged

chriscool added 4 commits April 30, 2016 07:13

package: use gx to import iptb

9b21478

License: MIT Signed-off-by: Christian Couder <chriscool@tuxfamily.org>

Replace existing iptb dependencies with the gx one

857fce1

Now that iptb has been added as a gx dependency, we can replace existing dependencies with the one from gx. License: MIT Signed-off-by: Christian Couder <chriscool@tuxfamily.org>

sharness/Makefile: check global dependencies

d4e7287

License: MIT Signed-off-by: Christian Couder <chriscool@tuxfamily.org>

test/Makefile: add global-deps target

13354f3

We need to make sure that go, gx, gx-go are properly installed. License: MIT Signed-off-by: Christian Couder <chriscool@tuxfamily.org>

chriscool force-pushed the use-gx-for-iptb branch from 7cf9153 to 13354f3 Compare April 30, 2016 05:27

chriscool added 2 commits April 30, 2016 08:39

package.json: import go-multihash

728e74c

License: MIT Signed-off-by: Christian Couder <chriscool@tuxfamily.org>

test/Makefile: use gx deps find for gx dependencies

5b17654

License: MIT Signed-off-by: Christian Couder <chriscool@tuxfamily.org>

chriscool force-pushed the use-gx-for-iptb branch from bda80be to 5b17654 Compare April 30, 2016 06:52

newer iptb package with correct deps

1770bad

License: MIT Signed-off-by: Jeromy <why@ipfs.io>

whyrusleeping merged commit f1df72b into ipfs:master May 3, 2016

chriscool mentioned this pull request May 4, 2016

Makefile: update gx and gx-go versions #2631

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use gx for iptb #2545

Use gx for iptb #2545

chriscool commented Apr 11, 2016

chriscool commented Apr 11, 2016

chriscool commented Apr 11, 2016

chriscool commented Apr 11, 2016

whyrusleeping commented Apr 12, 2016

chriscool commented Apr 15, 2016 •

edited

Loading

chriscool commented Apr 15, 2016

chriscool commented Apr 20, 2016

chriscool commented Apr 20, 2016

whyrusleeping commented Apr 20, 2016

chriscool commented Apr 23, 2016 •

edited

Loading

whyrusleeping commented Apr 23, 2016

chriscool commented Apr 30, 2016

chriscool commented Apr 30, 2016 •

edited

Loading

chriscool commented Apr 30, 2016

chriscool commented May 1, 2016

chriscool commented May 1, 2016

chriscool commented May 1, 2016 •

edited

Loading

chriscool commented May 1, 2016

whyrusleeping commented May 2, 2016

whyrusleeping commented May 2, 2016

chriscool commented May 2, 2016

whyrusleeping commented May 2, 2016

chriscool commented May 2, 2016

chriscool commented May 2, 2016

chriscool commented May 2, 2016

whyrusleeping commented May 2, 2016

whyrusleeping commented May 2, 2016

chriscool commented May 2, 2016

chriscool commented May 2, 2016

whyrusleeping commented May 2, 2016

chriscool commented May 2, 2016

whyrusleeping commented May 2, 2016

chriscool commented May 2, 2016

chriscool commented May 3, 2016

Use gx for iptb #2545

Use gx for iptb #2545

Conversation

chriscool commented Apr 11, 2016

chriscool commented Apr 11, 2016

chriscool commented Apr 11, 2016

chriscool commented Apr 11, 2016

whyrusleeping commented Apr 12, 2016

chriscool commented Apr 15, 2016 • edited Loading

chriscool commented Apr 15, 2016

chriscool commented Apr 20, 2016

chriscool commented Apr 20, 2016

whyrusleeping commented Apr 20, 2016

chriscool commented Apr 23, 2016 • edited Loading

whyrusleeping commented Apr 23, 2016

chriscool commented Apr 30, 2016

chriscool commented Apr 30, 2016 • edited Loading

chriscool commented Apr 30, 2016

chriscool commented May 1, 2016

chriscool commented May 1, 2016

chriscool commented May 1, 2016 • edited Loading

chriscool commented May 1, 2016

whyrusleeping commented May 2, 2016

whyrusleeping commented May 2, 2016

chriscool commented May 2, 2016

whyrusleeping commented May 2, 2016

chriscool commented May 2, 2016

chriscool commented May 2, 2016

chriscool commented May 2, 2016

whyrusleeping commented May 2, 2016

whyrusleeping commented May 2, 2016

chriscool commented May 2, 2016

chriscool commented May 2, 2016

whyrusleeping commented May 2, 2016

chriscool commented May 2, 2016

whyrusleeping commented May 2, 2016

chriscool commented May 2, 2016

chriscool commented May 3, 2016

chriscool commented Apr 15, 2016 •

edited

Loading

chriscool commented Apr 23, 2016 •

edited

Loading

chriscool commented Apr 30, 2016 •

edited

Loading

chriscool commented May 1, 2016 •

edited

Loading