WIP: Refactoring VirtualFS for usage in EFS and polykey

[CMCDragonkai]

Description

VirtualFS hasn't been updated for a years. It's not running on TypeScript (still using flow).

We may require to do some changes to support EFS. We need to make sure it still builds on our development platforms.

Tasks

1. Tests still work
2. Use the latest pkgs.nix and shell.nix which means an upgrade to the node and npm versions
3. Be able to run rollup to build distribution outputs
4. Add in TypeScript declarations (to avoid having to rewrite this all in TS)
5. Flow type checking still works?
6. Release new build

[CMCDragonkai]

All tests still pass which is great even on node v14.17.1.

However the rollup build fails for UMD builds:

[nix-shell:~/Projects/js-virtualfs]$ npm run rollup

> [email protected] rollup /home/cmcdragonkai/Projects/js-virtualfs
> rollup --config


lib/index.js → dist/index.node.es.js...
created dist/index.node.es.js in 1s

lib/index.js → dist/index.node.cjs.js...
created dist/index.node.cjs.js in 663ms

lib/index.js → dist/index.browser.umd.js...
[!] TypeError: The argument 'path' must be a string or Uint8Array without null bytes. Received '\x00commonjs-proxy:/home/cmcdragonkai/Projects/js-virtualfs/node_modules/core-js/library/modules/package.json'
TypeError [ERR_INVALID_ARG_VALUE] [ERR_INVALID_ARG_VALUE]: The argument 'path' must be a string or Uint8Array without null bytes. Received '\x00commonjs-proxy:/home/cmcdragonkai/Projects/js-virtualfs/node_modules/core-js/library/modules/package.json'
    at Object.readFile (fs.js:343:10)
    at next (/home/cmcdragonkai/Projects/js-virtualfs/node_modules/browser-resolve/index.js:98:12)
    at load_shims (/home/cmcdragonkai/Projects/js-virtualfs/node_modules/browser-resolve/index.js:116:7)
    at resolve (/home/cmcdragonkai/Projects/js-virtualfs/node_modules/browser-resolve/index.js:236:5)
    at /home/cmcdragonkai/Projects/js-virtualfs/node_modules/rollup-plugin-node-resolve/dist/rollup-plugin-node-resolve.cjs.js:66:5
    at new Promise (<anonymous>)
    at resolveId$1 (/home/cmcdragonkai/Projects/js-virtualfs/node_modules/rollup-plugin-node-resolve/dist/rollup-plugin-node-resolve.cjs.js:63:11)
    at /home/cmcdragonkai/Projects/js-virtualfs/node_modules/rollup/dist/rollup.js:1750:32

Seems like a weird error receiving null bytes.

The UMD build is needed in case to use VFS in browser. However it's not that important for PK since we most likely will need to provide a lot of polyfills to use in NativeScript and supply alternative implementations for a bunch of node native modules anyway. And we don't use PK in browsers.

[CMCDragonkai]

Probably just have to update rollup.

[CMCDragonkai]

I want to add these properties to the VirtualFSError.

path: string;
dest: string;

As these 2 parameters are available as parameters, they should be available as the parameters. Note that the message is then set here.

[CMCDragonkai]

That will allow me to map VirtualFSError to EncryptedFSError. It requires me to reconsruct it, and pass it as the object inheritance. To do so I have to reconstruct EncryptedFSError. It would be interesting to be able to dynamically chain an exception and make the exception inherit it though.. Anyway right now new EncryptedFSError('layer, errno[e.code], e.path, e.dest, e.syscall); would be ideal. That would be a standard way of mapping it into EFS error.

[CMCDragonkai]

I found that lseek and lseekSync currently doesn't return the current position.

It would be useful for these 2 functions to return the fd position. This can be useful for EFS to know where we are in the upperfs buffer.

As part of posix this is what is meant to be done anyway.

Right now they return void.

See: https://man7.org/linux/man-pages/man2/lseek.2.html#RETURN_VALUE

[CMCDragonkai]

Discovered a bug in js-virtualfs. It's current renameEntry doesn't unlink the inode if it is overwriting an existing name.

[CMCDragonkai]

Just updating rollup isn't going to work. There's quite a few things here that are outdated.

The only option is to remove the browser build and see if all works as usual.

Then afterwards to move to using TS and removing babel and rollup entirely.

[CMCDragonkai]

We lose the browser build, which means the README.md is a bit wrong. It only supports node usage. To be used inside a browser you have to bundle several polyfills:

1. buffer
2. stream
3. path
4. process

We can expect the end user to bring these in or we have to fix up our rollup configuration for the future.

[CMCDragonkai]

At any case it's better for the user of the library to designate their own overrides rather than expecting us to do it here.. it would be more flexible if not a bit more annoying.

[CMCDragonkai]

Another issue exists is that when setting an entry on to a directory inode, no check is made for an existing entry that if overwritten should induce a unlinking operation.

[CMCDragonkai]

Also https://gitlab.com/MatrixAI/Engineering/Polykey/js-virtualfs/-/issues/8 says that VFS cannot recursively delete directories.

But also because EFS is going to be more robustly tested than VFS, it's quite likely we won't actually be using VFS inside PK anywhere. So the priority for this PR is reduced.