Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Files over certain size don't seem to return correctly #132

Open
lastmjs opened this issue Jan 27, 2024 · 2 comments
Open

Files over certain size don't seem to return correctly #132

lastmjs opened this issue Jan 27, 2024 · 2 comments

Comments

@lastmjs
Copy link

lastmjs commented Jan 27, 2024

I haven't found a solution to this problem yet, but I've tracked the issue down to here: https://github.com/second-state/wasmedge-quickjs/blob/main/modules/internal/fs/stream.js#L175

In my Express app I'm trying to load a file that is just slightly bigger than the size of this buffer. I get this error:

"    at ERR_OUT_OF_RANGE (azle_main:6274)\n    at __node_internal_ (azle_main:6921)\n    at read (azle_main:20327)\n    at apply (native)\n    at <anonymous> (azle_main:7055)\n    at Promise (native)\n    at read (azle_main:7059)\n    at read (azle_main:21716)\n    at read (azle_main:19157)\n    at <anonymous> (azle_main:10176)\n    at resume_ (azle_main:9920)\n"

I will keep trying to track down a solution, but this is a major problem.

@lastmjs
Copy link
Author

lastmjs commented Jan 27, 2024

It seems like I have found the root cause of the issue. The file reading is supposed to be batch processed using buffers of length 16 * 1024, and the algorithm seems setup for that, but the parameters to this.file.read or very incorrect, leading to the batching algorithm not being used, so if you have a file over the original buffer size it will simply fail.

This is where the code needs to be changed: https://github.com/second-state/wasmedge-quickjs/blob/main/modules/internal/fs/stream.js#L218

Here's what I've done so far that seems to fix the issue:

const preLength = opts.end - opts.start - curPos + 1;

const { bytesRead: n } = await this.file.read(buffer, 0, preLength > buffer.byteLength ? buffer.byteLength : preLength, curPos === 0 ? opts.start : opts.start + curPos);

@lastmjs
Copy link
Author

lastmjs commented Jan 27, 2024

It seems that this code has not been well-tested...are there not Node.js or other test suites available to test against?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant