feat: getPath with carScope

[olizilla]

add getPath method as a generator that returns blocks for the targeted dag and all blocks traversed while resolving a cid+path string

supports carScope to specify what blocks to return for the resolved dag

• 'all': return the entire dag starting at path. (default)
• 'block': return the block identified by the path.
• 'file': Mimic gateway semantics: Return All blocks for a multi-block file or just enough blocks to enumerate a dir/map but not the dir contents.

see: storacha/freeway#33
see: storacha/freeway#34
see: ipfs/specs#402

TODO:

• find out how to identify the boundaries of a unixfs hamt

...unixfs-exporter seems to define it as "not having an empty or null Link.Name after the first 2 chars are stripped, which seems loose... what happens if the actual dir listing has 2 char long link names? see: https://github.com/ipfs/js-ipfs-unixfs/blob/e853049bd63d6773442e1540ae49b6a443ca8672/packages/ipfs-unixfs-exporter/src/resolvers/unixfs-v1/content/hamt-sharded-directory.ts#L20-L42

License: MIT

[olizilla]

@alanshaw plz to sanity check. also if you can point me at the right way to detect the boundary of the current hamt then that would save me some staring at things.

[olizilla]

turns out the swiss-army-knife has toString and fromString under multiformats/bytes! who knew!

[alanshaw]

LGTM

[alanshaw]

Is UnixFSEntry not already a Block?

[olizilla]

No, it lacks the all important bytes

[alanshaw]

...unixfs-exporter seems to define it as "not having an empty or null Link.Name after the first 2 chars are stripped, which seems loose... what happens if the actual dir listing has 2 char long link names?

You just strip the first two characters, and if the result is empty then it's a HAMT node, and if it is not empty it is not.

[olizilla]

is it the case that there is no edge case here because the user cannot create a dir with an um-named link.

[olizilla]

This should emit blocks not unixfs entries... where an entry is for a hamt we will only emit one thing, but we want to emit all the blocks that we traverse through the hamt. Also we don't want to emit a mix of unixfs entries and blocks.

[alanshaw]

Apologies I should have caught that in the review!

[olizilla]

@alanshaw ok the bits i missed first time around are now fixed, plz to eyeball

1. as you pointed out walkPath emits a single entry even for multi-block structures: fixed by passing in a custom blockstore that caches the traversed blocks so we can emit them all.
2. i was using the multiformats flavour block.links() function in my linkFilter, but that only understands ipld flavour link parsing, so dag-pb nodes link names are Links/<index>/Hash regardless of a Name field being present, so my hamtSearch wasn't working: fixed by special casing dagPB, which it would need to be anyway.

I've added tests for pathing through a sharded dir, as well as targeting a sharded dir as the end of the path as it's the most interesting case for carScope.

I've gone this root to ride on the work in unixfs-exporter for today, and get it working, hopefully without introducing any extra edge cases. I plan to remove this logic from here next week and put it in to https://github.com/web3-storage/ipfs-path which will become a block-centric rather than unixfsEntry-centric lib just for resolving ipfs paths... but first i want to land and ship verifiable paths and carScope to our gateway.

[alanshaw]

LGTM