hoverboard can run with env.CONTENT_CLAIMS url and will use content-claims when handling bitswap requests

[gobengo]

Motivation:

• https://github.com/web3-storage/secrets/issues/42

WIP

• test that hoverboard worker can be created with an env var to a content claims server, and then can read from that server
  • 2023-11-29 start
  • test creates a content-claims server, then passes server URL as env var for worker
  • test invokes hoverboard and asserts that hoverboard can fetch via contentClaimsClient
  • create a BlockStore that reads from content claims
  • use ContentClaimsBlockStore in hoverboard
  • test invokes hoverboard specifically via bitswap + helia and the helia can get whole DAG located via contentclaims
  • make freeway something hoverboard can depend on because there's too much good stuff in there to be copypastaing (and someone else might want to depend on freeway anyway) feat: tsconfig.json uses module=NodeNext, and package.json supports being imported freeway#95
• get feedback from @olizilla via code review here
• support more claims from https://github.com/web3-storage/content-claims/pull/44/files#
  • support length in relation claim not just offset
  • support range in location claim

[gobengo]

I don't think the workers-sdk testWorker fn that is unstable has a way of accessing the r2 bucket from outside the worker so I had to pul this in, and iiuc this new Miniflare 3 can only run on the built script in dist.

[travis]

This looks great to me! Would definitely feel comfier with another review from someone who's more familiar with this stuff, but I think this is mergable with or without the recommendations I've made.

[travis]

is there a reason to do this here rather than just doing const location = claim.location at the beginning of the transform function on line 182? it seems to me like it's just creating a bunch of extra datastructures, but wouldn't be surprised if I'm missing some nuance

[gobengo]

By doing them in separate streams, the second stream can process each location in parallel and also with separate error handling logic. instead of thinking about what to do if 2 locations are resolved but the 3rd gets an error or something.

[travis]

this pipe syntax keeps tripping me up, feels verbose, maybe this could be something like

.pipeThrough(transforms(
  async extractLocation (claim, controller) {
    for (const location of claim.location) { controller.enqueue([claim, location]) }
  },
  async fetchLocation ([claim, location], controller) {
    const response = await fetch(location, { headers: { accept: 'application/vnd.ipld.car' } })
    if (response.ok) {
      return controller.enqueue([claim, location, response])
    }
    throw new Error(`unexpected not-ok response ${response}`)
  },
  async readToBlocks ([claim, location, response], controller) {
    await response.body?.pipeThrough(new CARReaderStream()).pipeTo(new WritableStream({
      write: (block) => {
        // @todo validate that block.bytes hashes to `link`
        controller.enqueue({ claim, location, response, block })
      }
    }))
))

?

[gobengo]

where does transforms come from?

[gobengo]

I'm excited to use stream.compose in node but also trying here to use web streams and have as much as possible be easy to move to browsers. lmk if you know of a good library with those properties but personally I'd prefer not to make one rn.

[travis]

would love a good comment here summarizing what's going on - there's some complicated logic below and I'd love an abstract

[olizilla]

While reading this it struck me that it's a faff (programatically speaking) to have to unpack a relation claim + index + location claim to determine where to get a block from.

We can specify a location and a range in the location claim and have all the info we need to fetch the block bytes, so I'm making that work for materialised claims in storacha/content-claims#43

I'd like to land that and see if we can simplify things here.

[gobengo]

@olizilla sounds good I added these TODOs in the issue body above

[alanshaw]

superseded by #26