Switch from downloadjs to multi-download #174

ddelange · 2024-08-02T10:29:28Z

Version 3.0.0 ⚡️🚀

- docker-compose version 1.29.2, build unknown - Linux 5.15.0-1029-oracle blenderskool#35-Ubuntu SMP Tue Jan 24 15:17:52 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux ``` ERROR: The Compose file './docker-compose.yml' is invalid because: services.blaze-server.environment.TRUST_PROXY contains true, which is an invalid type, it should be a string, number, or a null ```

fix: docker-compose now

Version 3.0.1 🔨

Universal Analytics has been shut down

blenderskool · 2024-08-02T10:52:50Z

client/src/routes/App/FileTransfer/FileTransfer.js

+const download = async (file) => {
+  const a = document.createElement('a');
+  a.download = file.name;
+  a.href = URL.createObjectURL(file);
+  a.style.display = 'none';
+  document.body.append(a);
+  a.click();
+
+  // Chrome requires the timeout
+  await delay(100);
+  a.remove();
+};


Can we use downloadjs for this? From what I can see, delay of 1s is something that is to be added for fixing the issue. The individual file download seems to use same process as downloadjs

in addition to the 1s delay, to me it looks like this additional 100ms delay before removing the a again is not present in downloadjs. it looks like they immediately remove it after clicking: https://github.com/rndme/download/blob/09d6492f47ef18feca39c3d748960dce44f93a89/download.js#L116-L117

maybe chrome introduced the need for this additional short delay somewhere in the last 8 years (the latest release of downloadjs)?

I'd say this is a nice elimination of some legacy/unmaintained code, in exchange for a download function that's only 10 lines without dependencies

Ah I see the timeout. Changes seems fine to me, I'll also test it on my end in couple of browsers to ensure that it doesn't break existing flows.

OK! we might be able to reduce the 1000ms, I blamed it to a commit related to cross-origin URLs sindresorhus/multi-download@bf7580d#r144977818

client/src/routes/App/FileTransfer/FileTransfer.js

ddelange · 2024-08-02T12:20:32Z

client/src/routes/App/FileTransfer/FileTransfer.js

-            });
+            multiDownload(
+              // make regular File from webtorrent File https://github.com/webtorrent/webtorrent/blob/v2.4.14/lib/file.js#L7
+              files.map(file => new File([file.getBlob()], file.name, {type: file.type}))


btw, it looks like File.getBlob was removed in webtorrent v2.

this seems related to them switching to local storage in v2 as opposed to keeping complete torrents in memory in v1 ref webtorrent/webtorrent#86 (comment)

edit: they now have File.blob to load the whole torrent into memory and (more interestingly) File.streamURL (requires client.createServer to be ran beforehand).

Maybe blaze can even get rid of download.js altogether using the local storage functionality of webtorrent v2 🤔

~~they use FileSystemDirectoryHandle ref webtorrent/webtorrent#2208~~ this PR was superceded again

Blaze currently uses Webtorrent v1, and we'll need to upgrade to v2. From what I can recall since the last time I had checked, there were a bunch of API changes associated with v2 that we'd also have to make in Blaze which I wasn't fond of picking up then. Would you be interested in picking this up?

yeah, I'm also afraid it's a can of worms... the local storage feature might make it worthwhile though. I guess a 10GB file would now fill up 10GB of my swap right?

judging by the File API docs, I think using streamURL you can initiate a download before the torrent has finished downloading, and so download.js could theoretically be called upon as soon as the torrent is initiated.

most importantly, that way the file doesn't need to go fully into memory in a Blob in order to send to downloads.

file.getBlob and file.blob are functionally identical, it was simply renamed to match the W3C file API, rather than create our own structure, but getBlob also loaded the entire file into memory

ah just saw your cross comment, so file.streamURL is a viable option to get around the 'whole file in memory' pattern in the current downloading setup, with minimal overhead from client.createServer. that's great to hear and probably the way to move forward there!

the local storage feature might make it worthwhile though

@ddelange isn't the local storage limited to like 10MBs? I wonder how it would manage files of much higher sizes in such a limited space

in v2 the torrents directly download to /tmp/webtorrent by default using file system access.

path can be set on a per-torrent basis by the receiver before the torrent is added (folder selection is google chrome only if I understood the comments correctly)

in v2 the torrents directly download to /tmp/webtorrent by default using file system access.

for the sake of simplicity lets say sure, but if you want specifics, it actually writes to individual chunks via FSA's origin private file system, which functionally is no different from the likes of localStorage or IDB, except its a LOT faster, and allows for MUCH, MUCH, MUCH bigger file sizes, in extreme cases up to terabytes

on chrome that's a bit different, because IF YOU CHOOSE TO specify a rootDir, it can emulate the file structure in a specified folder as already mentioned, on top of saving chunks as cache to OPFS

that's the big difference for webtorrent v2, the storage and interfaces it provides:

createServer now provides insanely powerful, 0 overhead methods for rendering media like video, text, images using browser's own tools, instead of implementing custom tooling for it, which also allows other libraries to consume what webtorrent creates as its exposed via plain URL

storage isn't now memory, so we're no longer at the poor sub 2GB file limit, the most I could get webtorrent to store now is ~2TB after which chrome stopped enjoying handling so much data and started erroring

thanks for pointing to rootDir!

api.md points to storeOpts.rootDir but I think that should be opts.rootDir right?

based on the comments in api.md, I'm now not sure what's the difference between opts.path and opts.rootDir. should I ignore opts.path and only work with opts.rootDir?

ddelange · 2024-08-06T17:40:35Z

webtorrent v2 aside, @blenderskool did you get a chance to test more than 10 files?

blenderskool · 2024-08-06T17:54:05Z

@ddelange Not yet, will check it tomorrow. By the way, are you done with the changes for this PR?

ddelange · 2024-08-06T18:29:08Z

I'd say ready for review again yeah :)

ddelange · 2024-08-28T09:09:02Z

hi @blenderskool, did you get a chance to test more than 10 files with this fix?

ddelange · 2024-09-28T20:00:09Z

hi @blenderskool 👋 friendly reminder:)

blenderskool

@ddelange Finally got some time to test the changes. I don't know if it's just me but I see following errors with just a single file! Have you tested your changes?

client/src/routes/App/FileTransfer/FileTransfer.js

blenderskool · 2024-10-15T12:55:28Z

@ddelange I've pushed some fixes to your code and I can now see the multi-file(and single-file) download to work. Do test it from your end too if it is working as expected.

ddelange · 2024-10-15T13:14:06Z

awesome, your changes look clean!

client/src/utils/download.js

Co-authored-by: ddelange <[email protected]>

ddelange · 2024-10-28T08:43:33Z

ready for merge from my side 👍

blenderskool and others added 7 commits October 21, 2022 23:32

Delete FUNDING.yml

6fed0b5

Merge pull request blenderskool#157 from blenderskool/next

2494dea

Version 3.0.0 ⚡️🚀

Merge pull request blenderskool#161 from ruoduan-hub/master

f8acd96

fix: docker-compose now

Merge pull request blenderskool#170 from blenderskool/next

d9cbb0c

Version 3.0.1 🔨

Migrate to GA4

0e6f609

Universal Analytics has been shut down

Switch from downloadjs to multi-download

8b9e588

ddelange force-pushed the patch-1 branch from 4286e8c to 8b9e588 Compare August 2, 2024 10:42

blenderskool reviewed Aug 2, 2024

View reviewed changes

Move to utils/download.js

72de2fe

ddelange commented Aug 2, 2024

View reviewed changes

ddelange added 2 commits August 2, 2024 14:23

Amend comment

abe585a

Add URL.revokeObjectURL

9cd46a8

This was referenced Aug 2, 2024

Send files to downloads without client.createServer or file.blob webtorrent/webtorrent#2849

Closed

Upgrade to webtorrent v2 (avoid downloads in RAM) #175

Open

Simplify delay behaviour, add JSDoc comments

b11cee7

ddelange force-pushed the patch-1 branch from 26194d5 to b11cee7 Compare August 5, 2024 11:06

Typo

89ba327

blenderskool requested changes Oct 15, 2024

View reviewed changes

client/src/routes/App/FileTransfer/FileTransfer.js Show resolved Hide resolved

ddelange and others added 2 commits October 15, 2024 13:45

Fix typo

6f5c806

Fix multi-file downloads for WebTorrent

993e47b

blenderskool changed the base branch from master to next October 15, 2024 12:56

ddelange commented Oct 15, 2024

View reviewed changes

client/src/utils/download.js Outdated Show resolved Hide resolved

Update client/src/utils/download.js

a6ae197

Co-authored-by: ddelange <[email protected]>

blenderskool merged commit 2d34184 into blenderskool:next Oct 28, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Switch from downloadjs to multi-download #174

Switch from downloadjs to multi-download #174

ddelange commented Aug 2, 2024

blenderskool Aug 2, 2024

ddelange Aug 2, 2024

ddelange Aug 2, 2024

blenderskool Aug 2, 2024

ddelange Aug 2, 2024 •

edited

Loading

ddelange Aug 2, 2024 •

edited

Loading

blenderskool Aug 2, 2024

ddelange Aug 2, 2024

ddelange Aug 2, 2024 •

edited

Loading

ThaUnknown Aug 3, 2024

ddelange Aug 3, 2024

blenderskool Aug 3, 2024

ddelange Aug 3, 2024 •

edited

Loading

ThaUnknown Aug 3, 2024 •

edited

Loading

ddelange Aug 4, 2024

ddelange commented Aug 6, 2024

blenderskool commented Aug 6, 2024

ddelange commented Aug 6, 2024

ddelange commented Aug 28, 2024

ddelange commented Sep 28, 2024

blenderskool left a comment

blenderskool commented Oct 15, 2024

ddelange commented Oct 15, 2024

ddelange commented Oct 28, 2024

Switch from downloadjs to multi-download #174

Switch from downloadjs to multi-download #174

Conversation

ddelange commented Aug 2, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ddelange Aug 2, 2024 • edited Loading

Choose a reason for hiding this comment

ddelange Aug 2, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ddelange Aug 2, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ddelange Aug 3, 2024 • edited Loading

Choose a reason for hiding this comment

ThaUnknown Aug 3, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ddelange commented Aug 6, 2024

blenderskool commented Aug 6, 2024

ddelange commented Aug 6, 2024

ddelange commented Aug 28, 2024

ddelange commented Sep 28, 2024

blenderskool left a comment

Choose a reason for hiding this comment

blenderskool commented Oct 15, 2024

ddelange commented Oct 15, 2024

ddelange commented Oct 28, 2024

ddelange Aug 2, 2024 •

edited

Loading

ddelange Aug 2, 2024 •

edited

Loading

ddelange Aug 2, 2024 •

edited

Loading

ddelange Aug 3, 2024 •

edited

Loading

ThaUnknown Aug 3, 2024 •

edited

Loading