bitswap client #856
bitswap client #856
Conversation
* feat(booster-bitswap): booster bitswap MVP untested * refactor(booster-bitswap): use API for fetching blocks * fix(deps): update deps to compile * feat(booster-bitswap): makefile & fixes add commands to build booster-bitswap, and very a round tripped successful fetch from booster-bitswap * refactor: clean up unused vars etc * fix: booster-bitsawp - check error when creating libp2p key * refactor(node): avoid FreeAndUnsealed method Co-authored-by: Dirk McCormick <[email protected]> Co-authored-by: Anton Evangelatov <[email protected]>
* fix: return ipld ErrNotFound from remote blockstore interface * test: add more tests for ipld ErrNotFound
…bug in latest lotus (#802)
* feat(loadbalancer): add message types * feat(messages): add utility functions * feat(loadbalancer): initial load balancer impl implementation of the load balancer node itself * feat(loadbalancer): add service node implements code for running a service node * feat(loadbalancer): integrate into boost and booster-bitswap * Update loadbalancer/loadbalancer.go Co-authored-by: Rod Vagg <[email protected]> * Update loadbalancer/servicenode.go Co-authored-by: Rod Vagg <[email protected]> * Update loadbalancer/servicenode.go Co-authored-by: Rod Vagg <[email protected]> * Update loadbalancer/messages/messages.ipldsch Co-authored-by: Rod Vagg <[email protected]> * Update loadbalancer/messages/messages.ipldsch Co-authored-by: Rod Vagg <[email protected]> * refactor(loadbalancer): remove routing protocol remove the routing protocol, instead relying on a set config. also remove forwarding response for inbound requests * fix(loadbalancer): update tests * refactor(loadbalancer): integrate simplified load balancer removed pub keys to minimize network traffic, added api's to configure and update bitswap peer id, added auto config of bitswap peer id in booster-bitswap * docs(gen): regenerate api docs * chore(lint): fix lint errors * fix(loadbalancer): minor bridgestream fix * Update loadbalancer/servicenode.go Co-authored-by: dirkmc <[email protected]> * refactor(protocolproxy): address PR comments renames, reconfigured architecture, etc * refactor(make init print out peer id): remove apis and transparent peer id setting. have init print Co-authored-by: Rod Vagg <[email protected]> Co-authored-by: dirkmc <[email protected]>
* feat(booster-bitswap): add block filter via BadBits * refactor(booster-bitswap): use bitswap blockfilter for filtering * feat(blockfilter): only update when list is modified * feat(blockFilter): add on disk caching * Update cmd/booster-bitswap/blockfilter/blockfilter.go Co-authored-by: dirkmc <[email protected]> * fix(blockfilter): minor PR fixups Co-authored-by: dirkmc <[email protected]>
* chore(deps): upgrade to Lotus RC & libp2p v0.22 * chore(deps): update go to 1.18 * ci(circle): update circle to go 1.18 * style(imports): fix imports * fix(build): update ffi * fix(lint): fix deprecated strings.Title method * fix(mod): mod tidy
* refactor(booster-bitswap): minor UI fixes for booster-bitswap UI * Update cmd/booster-bitswap/init.go Co-authored-by: dirkmc <[email protected]> Co-authored-by: dirkmc <[email protected]>
dirkmc
force-pushed
the
feat/bitswap-client
branch
from
b022f1b to
efcfb9e
Compare
dirkmc
force-pushed
the
feat/bitswap-client
branch
2 times, most recently
from
6ccb1fa to
7ef9e1d
Compare
dirkmc
force-pushed
the
feat/bitswap-client
branch
2 times, most recently
from
43a79f6 to
65646ef
Compare
dirkmc
force-pushed
the
feat/bitswap-client
branch
from
65646ef to
0bd7736
Compare
There was a problem hiding this comment.
@dirkmc just a few pass-by comments on the fetch command in case you're interested.
Also worth noting that this code has been duplicated in a few places. Might be worth taking the pieces and putting them into a library so not everyone runs into the same issues/concerns although maybe this is different enough that the few hundred lines is no big deal.
Ones I know of:
- https://github.com/ipfs/ipget/
- https://github.com/application-research/estuary/blob/master/cmd/bsget/main.go
- https://github.com/ipfs-shipyard/vole/ (really just copied from above because I didn't want to deal with Filecoin FFI things and Go being uncooperative)
For example, all three implementations (and this one) have the same problem with host disconnects and requiring some code like the one in the linked issue that kubo has, to make larger data transfers more reliable even when the server is busy (maybe some peering/retry code should end up as a utility in go-libp2p 🤷).
|
|
||
| // setup libp2p host | ||
| log.Infow("generating libp2p key") | ||
| privKey, _, err := crypto.GenerateECDSAKeyPair(rand.Reader) |
There was a problem hiding this comment.
Any reason you went with ECDSA over ED25519, I tend to see the latter as the default in more places.
| libp2p.Transport(tcp.NewTCPTransport), | ||
| libp2p.Transport(quic.NewTransport), |
There was a problem hiding this comment.
Are these the only transports you're supporting because they're the only ones Boost supports?
| err = host.Connect(ctx, *serverAddrInfo) | ||
| if err != nil { | ||
| return fmt.Errorf("connecting to %s: %w", serverAddrInfo, err) |
There was a problem hiding this comment.
This will likely cause you problems with large files since if the connection gets pruned on the server side it'll never get re-established and your download will stall (e.g. like ipfs/ipget#103).
It's resolvable by trying to keep the connection alive.
|
@aschmahmann thanks a lot for the comments 🙏 This client is just intended as a convenience for testing, not to be used in production. So we kept it simple on purpose, but I agree that using kubo or of the other clients would make more sense in production. |
Fixes Allow direct connection to booster-bitswap
TODO: