Memory errors with large strings

[sswatson]

Recently I've been using DuckDB WASM to make a Chrome extension, and I've been getting memory access errors from the browser's WASM runtime whenever I try to insert a single row with a large-ish string (100KB, say) in a text field.

I can make a reproducible example if needed, but I'm curious if this is already a known issue with solutions or workarounds.

[sswatson]

In case it's helpful to others: the issue here turned out to be that queries around 2^17 bytes or longer seem to cause memory errors in general in DuckDB-WASM. Therefore, it's necessary to use the virtual file system features rather than large INSERT statements if you have more than a little bit of data. Once I switched to that approach, everything worked fine.

[dude0001]

@sswatson can you give an example of what worked for you?

[sswatson]

Sure, I don't have much time so I'm just going to copy the relevant section of code from my project to convey the gist -- I think it will be sufficient:

conn.query("create table if not exists saved_pages (visit_time timestamp, url text, html text);");
await db.registerFileText(
    'new_page.json',
    JSON.stringify([{
        visit_time: new Date().toISOString(),
        url,
        html,
    }]),
);
await conn.query('insert into saved_pages select current_timestamp, url, html from "new_page.json"');
await db.dropFile('new_page.json');

Previously I was doing that without registerFileText; I would just inline the contents of the string into the query. If you do that and the string gets too long, you'll run into problems.