Fantastic js-ctypes! select and fd_set

[Noitidart]

I was working on a bugzilla issue to bring a file watching API and needed to use select and needed the FD_SET macro. (Noitidart/jscFileWatcher#12) I found you guys already wrote up this macro for use with select here:

tenshi/angel-player/src/chrome/content/common/serport_posix.js

Line 497 in 9b32732

function fd_set_set(fdset, fd) {

This is real awesome thank you! May I "borrow" this please haha

[Noitidart]

PS I saw the note "this should hopefully work" which kind of worries me:

• tenshi/angel-player/src/chrome/content/common/serport_posix.js

  Line 474 in 9b32732

  // We have an array of int32. This should hopefully work on Darwin

• tenshi/angel-player/src/chrome/content/common/serport_posix.js

  Line 437 in 9b32732

  // figure out which byte we should be in. We assume a 64-bit platform

I'm not intending to use this on mac but it would be awesome if we can improve this to not have any assumptions. Can you please mention to me in what direction I can go to make these not based on any assumptions, like I can calculate if the system is 64bit or not with no assumption with:

if (ctypes.voidptr_t.size === 4 /* 32-bit */) {
    var is64bit = false;
} else if (ctypes.voidptr_t.size === 8 /* 64-bit */) {
    var is64bit = true;
} else {
    throw new Error('huh??? not 32 or 64 bit?!?!');
}

[Noitidart]

Here is how i'm using it: https://github.com/Noitidart/jscFileWatcher/blob/inotify-move-to-select/modules/workers/FSWPollWorker.js#L662

[zentner-kyle]

Hi Noitidart,

Unfortunately I'm very busy the next couple of days, so I can't help
much. PiE also isn't really using this code any more. But, I know that
sizeof(void *) != sizeof(long) on many platforms. I'm pretty sure the
easiest way of handling this on most platforms it to assume that the
fd_set is just a fully packed array (of e.g. bytes), and compute the
result of that function as

return {'elem8': Math.floor(fd / 8), 'bitpos8': fd % 8};

That looks like it will run on most Unix systems based on my reading of
http://fxr.watson.org/fxr/source/sys/fd_set.h?v=NETBSD
and
http://fxr.watson.org/fxr/source/sys/select.h
and
http://fxr.watson.org/fxr/source/include/linux/posix_types.h?v=linux-2.6#L38
and
https://developer.apple.com/library/mac/documentation/Darwin/Reference/ManPages/man5/types.5.html
and
http://fxr.watson.org/fxr/source/sys/select.h?v=OPENBSD#L61
and
http://fxr.watson.org/fxr/source/include/sys/select.h?v=minix-3-1-1#L33

since all of those systems put the array at the beginning of the fd_set
struct.

However, it's not guaranteed to work according to the POSIX standard,
and I can't confirm that it will work on Solaris or some other targets
(newlib?, cygwin?).

It definitely won't work on Windows, as you can see from the layout
listed on this page:
https://msdn.microsoft.com/en-us/library/windows/desktop/ms737873%28v=vs.85%29.aspx

However, it should be relatively easy to get around that (just add the
sizeof(u_int) to the elem8 field). Let me know if you have any questions!
I recently discovered how useful file watching can be, so I'd love to
see your extension completed.

[Noitidart]

Thank you very much @zentner-kyle for that input! I'll definitely use that and share any improvements we make. As of right I dropped your code into mine and it's working so thank you very much for that! :)

I noticed I had to do fd_set every time in the loop. Also would it be safe to cache the value of elem8 and bitpos8 in a session? It's weird cuz I've noticed it change but I can't recall what was different except session.

[Noitidart]

Hey @zentner-kyle we have OS.Constants.Sys.bits available to us now :) It tells us if its 32 or 64 bits :)

I wanted to rewrite this so it works perfectly but I was wondering if you had time :p I was in middle of some open source stuff but after I finish that Ill definitely revisit this if you haven't knocked it out by then :) A sincere thanks again for sharing this awesomeness in the open!

[Noitidart]

Hi @zentner-kyle I'm wrapping up my file watcher, I converted it to a submodule so it can be included into any addon, and possibly drop it into Firefox itself - https://bugzilla.mozilla.org/show_bug.cgi?id=958280

I was wondering your thoughts on this FD_XXX macros, an awesome person from Stackoverflow provided this. Would this accomdate all situations (mac 32bit, mac 64bit, *nix 32bit, *nix 64bit, *nix 64bit but 32bit)?

                var SIZE_32 = 4
                var SIZE_64 = 8

                var LITTLE_ENDIAN = [0, 1, 2, 3, 4, 5, 6, 7];
                var BIG_ENDIAN = [7, 6, 5, 4, 3, 2, 1, 0];

                function fdset(setSize, endianness, wordSize)
                {
                                var size = wordSize
                    var buffer = new Uint8Array(div(setSize + 7, 8));



                    function div(a, b)
                    {
                        return Math.floor(a / b);
                    }

                    function make_index(index)
                    {
                        return div(index, 8 * size) * size + endianness[div(index % (8 * size), 8)] % size;
                    }

                    buffer.set_bit = function(index)
                    {
                        buffer[make_index(index)] |= 1 << (index % 8);
                    };

                    buffer.clear_bit = function(index)
                    {
                        buffer[make_index(index)] &= ~(index % 8);
                    };

                    buffer.get_bit = function(index)
                    {
                        return buffer[make_index(index)] & 1 << (index % 8);
                    };

                    buffer.zero = function()
                    {
                        buffer.fill(0);
                    }


                    return buffer;
                }

                function FD_SET(fd, fdset)
                {
                    fdset.set_bit(fd);
                }

                function FD_ISSET(fd, fdset)
                {
                    return !!fdset.get_bit(fd);
                }

                function FD_CLR(fd, fdset)
                {
                    return fdset.clear_bit(fd);
                }

                function FD_ZERO(fdset)
                {
                    return fdset.zero();
                }

            var s = fdset(128, LITTLE_ENDIAN, SIZE_64);

            //s in an Uint8Array

            console.log(s);

            FD_SET(0, s);    //Byte 0 = 1
            FD_SET(9, s);    //Byte 1 = 2
            FD_SET(18, s);   //Byte 2 = 4
            FD_SET(27, s);   //Byte 3 = 8
            FD_SET(36, s);   //Byte 4 = 16
            FD_SET(45, s);   //Byte 5 = 32
            FD_SET(54, s);   //Byte 6 = 64
            FD_SET(63, s);   //Byte 7 = 128

            FD_SET(120, s);  //Byte 15 = 1
            FD_SET(113, s);  //Byte 14 = 2
            FD_SET(106, s);  //Byte 13 = 4
            FD_SET(99, s);   //Byte 12 = 8
            FD_SET(92, s);   //Byte 11 = 16
            FD_SET(85, s);   //Byte 10 = 32
            FD_SET(78, s);   //Byte 9 = 64
            FD_SET(71, s);   //Byte 8 = 128

            console.log(s);

            //64 bits, BE: [128, 64, 32, 16, 8, 4, 2, 1, 1, 2, 4, 8, 16, 32, 64, 128]
            //64 bits, LE: [1, 2, 4, 8, 16, 32, 64, 128, 128, 64, 32, 16, 8, 4, 2, 1]
            //32 bits, BE: [8, 4, 2, 1, 128, 64, 32, 16, 16, 32, 64, 128, 1, 2, 4, 8]
            //32 bits, LE: [1, 2, 4, 8, 16, 32, 64, 128, 128, 64, 32, 16, 8, 4, 2, 1]

And to detect endianess would this be suitable:

function isLittleEndian() {
    var buffer = new ArrayBuffer(2);
    new DataView(buffer).setInt16(0, 256, true);
    return new Int16Array(buffer)[0] === 256;
};

And then for wordSize would this be suitable? https://developer.mozilla.org/en-US/docs/Mozilla/js-ctypes/js-ctypes_reference/ctypes#Types_whose_size_varies_depending_on_platform

wordSize = ctypes.size_t.size; // on 64bit windows with 32bit firefox this is 4. on 64bit windows with 64bit firefox this is 8.

[zentner-kyle]

Hi @Noitidart,

Sorry for the slow response. I kinda get a lot of Github notifications, and this one was lost in the flood.
This looks pretty good. I would also like to see the Mozilla DevTools gain file watching ability, so it would be great if you could land this capability there. It might mostly work as is, but there are a handful of minor issues.

I think the only issue you need to address is that fd_set is assumed by native code to have a specific size. If you allocate an array smaller than that, and pass it to select as an fd_set, you can segfault the process you're running in.

Unfortunately, the size varies between each Unix. Fortunately, being too large should not be a problem. As far as I can tell, the largest fd_set variant is that used by Linux, which is 1024 bits. This variant is used by almost every Unix, excluding at least NetBSD and Minix (both of which use a smaller size). So if you always make setSize 128 or greater, we can avoid crashing.

Unforunately, there's go guarantee that wordSize is correct in general. From reading the headers, it is correct for x86 Linux and x86_64 Linux. It is probably also correct for Linux on other platforms (like ARM), but I'm not positive. However, it appears that OSX and several of the BSD's always uses 32 bit integers for each entry in fd_set. Fortunately, if this value is wrong, the result only changes on big endian architectures, and most processors are little endian these days.

Besides that, it looks like there might be a bug in the clear_bit function, since it doesn't shift by the index. I think the correct function would be:

buffer.clear_bit = function(index) {
  buffer[make_index(index)] &= ~(1 << (index % 8));
};

If you'd like to point me at a repo to review instead, I'd be happy to take a look at that.

[Noitidart]

Thanks very much @zentner-kyle - I really appreciate that background and checking. Especially the platform dependent explanation.

The repo is here - https://github.com/Noitidart/jscFileWatcher/

It is meant for use from a ChromeWorker instance, here is live demo usage - https://github.com/Noitidart/CommPlayground/tree/jscfilewatcher-demo

The non-setup code is here - https://github.com/Noitidart/CommPlayground/blob/jscfilewatcher-demo/resources/scripts/MainWorker.js

There is some setup on the mainthread ( https://github.com/Noitidart/CommPlayground/blob/jscfilewatcher-demo/bootstrap.js ) as Gio file watcher requires callback on mainthread (from what I could gather).

Right now (master commit of jscFileWatcher module) I'm forcing it to use inotify on all linux as i'm trying to fix the poll there. I really badly wanted to use select but I'm not sure if I'm getting it right. So I'm using poll right now - https://github.com/Noitidart/jscFileWatcher/blob/master/DirectoryWatcherPollWorker.js#L418

For inotify, I'm getting an unexplained EINTR ( https://bugzilla.mozilla.org/show_bug.cgi?id=1288293 ), but everything is almost set. I just have to fix this EINTR issue, and then make the delivery to the callback set up the by the developer in a common way across all platforms. Right now you will see it successfully trigger callback on all Windows, OSX (10.7), GTK (if inotify is not forced) systems.

I really appreciate your input!