Skip to content
This repository was archived by the owner on Jun 20, 2024. It is now read-only.

feat: IPIP-402 based backpressure #160

Merged
merged 49 commits into from
Aug 15, 2023
Merged

feat: IPIP-402 based backpressure #160

Show file tree
Hide file tree
Changes from 31 commits
Commits
Show all changes
49 commits
Select commit Hold shift + click to select a range
83eec5d
chore: update boxo
aschmahmann Jun 9, 2023
ce24a3f
feat: switch GetCAR to be backpressured
aschmahmann Jun 10, 2023
7ec385a
feat: switch ResolvePath, Head and GetBlock to be backpressured
aschmahmann Jun 12, 2023
468796a
feat: switch Get to be backpressured (not fully implemented)
aschmahmann Jun 12, 2023
db4ecfe
switch fetch to do escaping
aschmahmann Jun 29, 2023
2458e3e
fix detection of go-ipld-prime schema errors and path resolution
aschmahmann Jul 6, 2023
8b2cdc7
fix boxo go.mod
aschmahmann Jul 6, 2023
56fce88
fix passing range bytes in CAR requests, and reporting entity ranges
aschmahmann Jul 6, 2023
965e29d
chore: remove unused code
aschmahmann Jul 6, 2023
c2fb61a
feat: add support for backpressured tar files
aschmahmann Jul 13, 2023
0691af9
chore: removed unused testing tools
aschmahmann Jul 13, 2023
7f8eb45
feat: add some partial traversal tests and fixes
aschmahmann Jul 18, 2023
60f2d4a
fix: HAMT fetching bug for Get requests
aschmahmann Jul 21, 2023
1c28ef4
test: add tests for Get on HAMT directories
aschmahmann Jul 21, 2023
f473fea
feat: fix and add tests for Get on large files. Consolidate some Get …
aschmahmann Jul 24, 2023
5a99a19
feat: allow passing a custom prometheus registry to the graph gateway…
aschmahmann Jul 24, 2023
d7383de
fix: remove unneeded block fetching remnants from graph backend. also…
aschmahmann Jul 24, 2023
79523e7
fix: resolve issue with file resumption on error and add test
aschmahmann Jul 24, 2023
5e44660
chore: remove unneeded blocks pubsub implementation
aschmahmann Jul 24, 2023
c0f250e
fix: TestGetHAMTDirectory is no longer flaky, although it is more imp…
aschmahmann Jul 25, 2023
be81dfc
fix: preserve '/' characters even after escaping to help URL parsers …
aschmahmann Jul 25, 2023
c3fb8d8
refactor: add contentPathToCarUrl and tests
lidel Jul 26, 2023
7691b62
bump boxo dependency to handle _redirects better and switch from boxo…
aschmahmann Jul 27, 2023
236009c
update caboose to fix issues draining responses
aschmahmann Jul 27, 2023
53101eb
add more information when erroring on receiving an unexpected cid
aschmahmann Jul 27, 2023
7df26b0
fix: switch use of ErrPartialResponse to not use pointers so they wor…
aschmahmann Jul 27, 2023
b1e8074
test: add test for file range requests with errors during download
aschmahmann Jul 27, 2023
b3d2909
fix: pass through range parameters for partial file load resumption
aschmahmann Jul 27, 2023
03cba57
fix: perform gateway error conversion for graph backend
aschmahmann Jul 31, 2023
28801f0
feat: add ipip-412 parameters for order and duplicates to non-saturn …
aschmahmann Jul 31, 2023
db66484
fix: allow reads with empty car roots (#182)
willscott Aug 3, 2023
8c217f9
fix memory leak
aarshkshah1992 Aug 7, 2023
06077bc
update to latest caboose
willscott Aug 8, 2023
0bbc0c4
Send verification error to Caboose (#184)
aarshkshah1992 Aug 8, 2023
addb2a1
Merge remote-tracking branch 'origin/main' into feat/backpressure
willscott Aug 9, 2023
4958149
Merge branch 'feat/backpressure' of github.com:ipfs/bifrost-gateway i…
willscott Aug 9, 2023
c9fd369
bump go-unixfsnode and enable returning the sizes of items in HAMT di…
aschmahmann Aug 10, 2023
e2be930
return gateway errors on CAR fetching failure
aschmahmann Aug 10, 2023
d8485e2
add verifcid check
aschmahmann Aug 10, 2023
1c54c91
test: run conformance with GRAPH_BACKEND=true
lidel Aug 14, 2023
a978e34
fix: correctly decode UnixFS symlinks
aschmahmann Aug 15, 2023
4c5e7f9
fix: return starting at the correct offset when doing a Get request t…
aschmahmann Aug 15, 2023
ce2ef89
fix: add identity-blockstore style behavior to carToLinearBlockGetter
aschmahmann Aug 15, 2023
1275db5
chore: add guard to GetCar so we don't accidentally forget to handle …
aschmahmann Aug 15, 2023
e605576
feat: handle IPIP-412 requests with and without duplicates, backed by…
aschmahmann Aug 15, 2023
c23e66f
ci: use a kubo development commit for conformance tests
aschmahmann Aug 15, 2023
983ad96
test: fix TestGetHAMTDirectory to not look for //index.html
aschmahmann Aug 15, 2023
004ea10
test: fix CAR retry tests to use CarWriter that allows duplicate blocks
aschmahmann Aug 15, 2023
5325e53
Merge branch 'main' into feat/backpressure
aschmahmann Aug 15, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 2 additions & 34 deletions blockstore.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,14 +3,11 @@ package main
import (
"context"
"errors"
"fmt"
"strings"
"time"

"github.com/filecoin-saturn/caboose"
"github.com/ipfs/bifrost-gateway/lib"
blockstore "github.com/ipfs/boxo/blockstore"
exchange "github.com/ipfs/boxo/exchange"
"github.com/ipfs/boxo/gateway"
blocks "github.com/ipfs/go-block-format"
"github.com/ipfs/go-cid"
"go.uber.org/zap/zapcore"
Expand Down Expand Up @@ -38,7 +35,7 @@ func (e *exchangeBsWrapper) GetBlock(ctx context.Context, c cid.Cid) (blocks.Blo

blk, err := e.bstore.Get(ctx, c)
if err != nil {
return nil, gatewayError(err)
return nil, lib.GatewayError(err)
}
return blk, nil
}
Expand Down Expand Up @@ -67,33 +64,4 @@ func (e *exchangeBsWrapper) Close() error {
return nil
}

// gatewayError translates underlying blockstore error into one that gateway code will return as HTTP 502 or 504
// it also makes sure Retry-After hint from remote blockstore will be passed to HTTP client, if present.
func gatewayError(err error) error {
if errors.Is(err, &gateway.ErrorStatusCode{}) ||
errors.Is(err, &gateway.ErrorRetryAfter{}) {
// already correct error
return err
}

// All timeouts should produce 504 Gateway Timeout
if errors.Is(err, context.DeadlineExceeded) ||
errors.Is(err, caboose.ErrSaturnTimeout) ||
// Unfortunately this is not an exported type so we have to check for the content.
strings.Contains(err.Error(), "Client.Timeout exceeded") {
return fmt.Errorf("%w: %s", gateway.ErrGatewayTimeout, err.Error())
}

// (Saturn) errors that support the RetryAfter interface need to be converted
// to the correct gateway error, such that the HTTP header is set.
for v := err; v != nil; v = errors.Unwrap(v) {
if r, ok := v.(interface{ RetryAfter() time.Duration }); ok {
return gateway.NewErrorRetryAfter(err, r.RetryAfter())
}
}

// everything else returns 502 Bad Gateway
return fmt.Errorf("%w: %s", gateway.ErrBadGateway, err.Error())
}

var _ exchange.Interface = (*exchangeBsWrapper)(nil)
10 changes: 8 additions & 2 deletions blockstore_proxy.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,14 +43,20 @@ func (ps *proxyBlockStore) Fetch(ctx context.Context, path string, cb lib.DataCa
return err
}
goLog.Debugw("car fetch", "url", req.URL)
req.Header.Set("Accept", "application/vnd.ipld.car")
req.Header.Set("Accept", "application/vnd.ipld.car;order=dfs;dups=y")
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note: this resolves #177 for non-Saturn backends. Caboose/Saturn will have to change when they're ready.

resp, err := ps.httpClient.Do(req)
if err != nil {
return err
}

if resp.StatusCode != http.StatusOK {
return fmt.Errorf("http error from car gateway: %s", resp.Status)
errData, err := io.ReadAll(resp.Body)
if err != nil {
err = fmt.Errorf("could not read error message: %w", err)
} else {
err = fmt.Errorf("%q", string(errData))
}
return fmt.Errorf("http error from car gateway: %s: %w", resp.Status, err)
}

err = cb(path, resp.Body)
Expand Down
44 changes: 0 additions & 44 deletions blockstore_test.go
View file
Open in desktop

This file was deleted.

54 changes: 8 additions & 46 deletions go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,25 +3,23 @@ module github.com/ipfs/bifrost-gateway
go 1.19

require (
github.com/cskr/pubsub v1.0.2
github.com/filecoin-saturn/caboose v0.0.3
github.com/filecoin-saturn/caboose v0.0.3-no-response-drain-2
github.com/hashicorp/go-multierror v1.1.1
github.com/hashicorp/golang-lru/v2 v2.0.1
github.com/ipfs/boxo v0.11.0
github.com/ipfs/boxo v0.11.1-0.20230803093408-3b83506a2ea5
github.com/ipfs/go-block-format v0.1.2
github.com/ipfs/go-cid v0.4.1
github.com/ipfs/go-ipld-format v0.5.0
github.com/ipfs/go-log/v2 v2.5.1
github.com/ipfs/go-unixfsnode v1.7.1
github.com/ipld/go-car v0.6.0
github.com/ipfs/go-unixfsnode v1.7.3-0.20230718163022-62f4a1c8d46e
github.com/ipld/go-car v0.6.2
github.com/ipld/go-car/v2 v2.10.2-0.20230622090957-499d0c909d33
github.com/ipld/go-codec-dagpb v1.6.0
github.com/ipld/go-ipld-prime v0.20.0
github.com/libp2p/go-libp2p v0.26.3
github.com/libp2p/go-libp2p-kad-dht v0.23.0
github.com/libp2p/go-libp2p-routing-helpers v0.7.0
github.com/mitchellh/go-server-timing v1.0.1
github.com/multiformats/go-multicodec v0.9.0
github.com/multiformats/go-multihash v0.2.3
github.com/prometheus/client_golang v1.15.1
github.com/rs/dnscache v0.0.0-20211102005908-e0241e321417
github.com/spf13/cobra v1.6.1
Expand All @@ -33,117 +31,83 @@ require (
go.opentelemetry.io/otel/sdk v1.14.0
go.opentelemetry.io/otel/trace v1.14.0
go.uber.org/atomic v1.10.0
go.uber.org/multierr v1.9.0
go.uber.org/zap v1.24.0
)

require (
github.com/alecthomas/units v0.0.0-20211218093645-b94a6e3cc137 // indirect
github.com/asecurityteam/rolling v0.0.0-20230418204413-b4052899307d // indirect
github.com/benbjohnson/clock v1.3.0 // indirect
github.com/beorn7/perks v1.0.1 // indirect
github.com/cenkalti/backoff/v4 v4.2.0 // indirect
github.com/cespare/xxhash/v2 v2.2.0 // indirect
github.com/containerd/cgroups v1.0.4 // indirect
github.com/coreos/go-systemd/v22 v22.5.0 // indirect
github.com/crackcomm/go-gitignore v0.0.0-20170627025303-887ab5e44cc3 // indirect
github.com/davecgh/go-spew v1.1.1 // indirect
github.com/davidlazar/go-crypto v0.0.0-20200604182044-b73af7476f6c // indirect
github.com/decred/dcrd/dcrec/secp256k1/v4 v4.1.0 // indirect
github.com/docker/go-units v0.5.0 // indirect
github.com/dustin/go-humanize v1.0.0 // indirect
github.com/elastic/gosigar v0.14.2 // indirect
github.com/felixge/httpsnoop v1.0.3 // indirect
github.com/flynn/noise v1.0.0 // indirect
github.com/francoispqt/gojay v1.2.13 // indirect
github.com/gabriel-vasile/mimetype v1.4.1 // indirect
github.com/go-logr/logr v1.2.3 // indirect
github.com/go-logr/stdr v1.2.2 // indirect
github.com/go-task/slim-sprig v0.0.0-20210107165309-348f09dbbbc0 // indirect
github.com/godbus/dbus/v5 v5.1.0 // indirect
github.com/gogo/protobuf v1.3.2 // indirect
github.com/golang/gddo v0.0.0-20180823221919-9d8ff1c67be5 // indirect
github.com/golang/mock v1.6.0 // indirect
github.com/golang/protobuf v1.5.3 // indirect
github.com/google/gopacket v1.1.19 // indirect
github.com/google/pprof v0.0.0-20221203041831-ce31453925ec // indirect
github.com/google/uuid v1.3.0 // indirect
github.com/grpc-ecosystem/grpc-gateway/v2 v2.7.0 // indirect
github.com/hashicorp/errwrap v1.1.0 // indirect
github.com/hashicorp/go-multierror v1.1.1 // indirect
github.com/hashicorp/golang-lru v0.5.4 // indirect
github.com/huin/goupnp v1.0.3 // indirect
github.com/inconshreveable/mousetrap v1.0.1 // indirect
github.com/ipfs/bbloom v0.0.4 // indirect
github.com/ipfs/go-bitfield v1.1.0 // indirect
github.com/ipfs/go-blockservice v0.5.0 // indirect
github.com/ipfs/go-datastore v0.6.0 // indirect
github.com/ipfs/go-ipfs-blockstore v1.3.0 // indirect
github.com/ipfs/go-ipfs-delay v0.0.1 // indirect
github.com/ipfs/go-ipfs-ds-help v1.1.0 // indirect
github.com/ipfs/go-ipfs-exchange-interface v0.2.0 // indirect
github.com/ipfs/go-ipfs-redirects-file v0.1.1 // indirect
github.com/ipfs/go-ipfs-util v0.0.2 // indirect
github.com/ipfs/go-ipld-cbor v0.0.6 // indirect
github.com/ipfs/go-ipld-legacy v0.2.1 // indirect
github.com/ipfs/go-libipfs v0.6.0 // indirect
github.com/ipfs/go-log v1.0.5 // indirect
github.com/ipfs/go-merkledag v0.11.0 // indirect
github.com/ipfs/go-metrics-interface v0.0.1 // indirect
github.com/ipfs/go-verifcid v0.0.2 // indirect
github.com/jackpal/go-nat-pmp v1.0.2 // indirect
github.com/jbenet/go-temp-err-catcher v0.1.0 // indirect
github.com/jbenet/goprocess v0.1.4 // indirect
github.com/klauspost/compress v1.15.15 // indirect
github.com/klauspost/cpuid/v2 v2.2.3 // indirect
github.com/koron/go-ssdp v0.0.3 // indirect
github.com/libp2p/go-buffer-pool v0.1.0 // indirect
github.com/libp2p/go-cidranger v1.1.0 // indirect
github.com/libp2p/go-doh-resolver v0.4.0 // indirect
github.com/libp2p/go-flow-metrics v0.1.0 // indirect
github.com/libp2p/go-libp2p-asn-util v0.2.0 // indirect
github.com/libp2p/go-libp2p-kad-dht v0.23.0 // indirect
github.com/libp2p/go-libp2p-kbucket v0.5.0 // indirect
github.com/libp2p/go-libp2p-record v0.2.0 // indirect
github.com/libp2p/go-msgio v0.3.0 // indirect
github.com/libp2p/go-nat v0.1.0 // indirect
github.com/libp2p/go-netroute v0.2.1 // indirect
github.com/libp2p/go-reuseport v0.2.0 // indirect
github.com/libp2p/go-yamux/v4 v4.0.0 // indirect
github.com/marten-seemann/tcp v0.0.0-20210406111302-dfbc87cc63fd // indirect
github.com/mattn/go-isatty v0.0.17 // indirect
github.com/matttproud/golang_protobuf_extensions v1.0.4 // indirect
github.com/miekg/dns v1.1.50 // indirect
github.com/mikioh/tcpinfo v0.0.0-20190314235526-30a79bb1804b // indirect
github.com/mikioh/tcpopt v0.0.0-20190314235656-172688c1accc // indirect
github.com/minio/sha256-simd v1.0.0 // indirect
github.com/mr-tron/base58 v1.2.0 // indirect
github.com/multiformats/go-base32 v0.1.0 // indirect
github.com/multiformats/go-base36 v0.2.0 // indirect
github.com/multiformats/go-multiaddr v0.8.0 // indirect
github.com/multiformats/go-multiaddr-dns v0.3.1 // indirect
github.com/multiformats/go-multiaddr-fmt v0.1.0 // indirect
github.com/multiformats/go-multibase v0.2.0 // indirect
github.com/multiformats/go-multihash v0.2.3 // indirect
github.com/multiformats/go-multistream v0.4.1 // indirect
github.com/multiformats/go-varint v0.0.7 // indirect
github.com/onsi/ginkgo/v2 v2.5.1 // indirect
github.com/opencontainers/runtime-spec v1.0.3-0.20211123151946-c2389c3cb60a // indirect
github.com/opentracing/opentracing-go v1.2.0 // indirect
github.com/openzipkin/zipkin-go v0.4.1 // indirect
github.com/patrickmn/go-cache v2.1.0+incompatible // indirect
github.com/pbnjay/memory v0.0.0-20210728143218-7b4eea64cf58 // indirect
github.com/petar/GoLLRB v0.0.0-20210522233825-ae3b015fd3e9 // indirect
github.com/pkg/errors v0.9.1 // indirect
github.com/pmezard/go-difflib v1.0.0 // indirect
github.com/polydawn/refmt v0.89.0 // indirect
github.com/prometheus/client_model v0.3.0 // indirect
github.com/prometheus/common v0.42.0 // indirect
github.com/prometheus/procfs v0.9.0 // indirect
github.com/quic-go/qpack v0.4.0 // indirect
github.com/quic-go/qtls-go1-19 v0.2.1 // indirect
github.com/quic-go/qtls-go1-20 v0.1.1 // indirect
github.com/quic-go/quic-go v0.33.0 // indirect
github.com/quic-go/webtransport-go v0.5.2 // indirect
github.com/raulk/go-watchdog v1.3.0 // indirect
github.com/serialx/hashring v0.0.0-20200727003509-22c0c7ab6b1b // indirect
github.com/spaolacci/murmur3 v1.1.0 // indirect
github.com/spf13/pflag v1.0.5 // indirect
Expand All @@ -167,8 +131,7 @@ require (
go.opentelemetry.io/otel/exporters/zipkin v1.14.0 // indirect
go.opentelemetry.io/otel/metric v0.37.0 // indirect
go.opentelemetry.io/proto/otlp v0.19.0 // indirect
go.uber.org/dig v1.15.0 // indirect
go.uber.org/fx v1.18.2 // indirect
go.uber.org/multierr v1.9.0 // indirect
golang.org/x/crypto v0.6.0 // indirect
golang.org/x/exp v0.0.0-20230213192124-5e25df0256eb // indirect
golang.org/x/mod v0.7.0 // indirect
Expand All @@ -184,5 +147,4 @@ require (
google.golang.org/protobuf v1.30.0 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect
lukechampine.com/blake3 v1.1.7 // indirect
nhooyr.io/websocket v1.8.7 // indirect
)
Loading