sodium-native issues

[staltz]

I'm quite close to getting sodium-native (libsodium) working on nodejs-mobile, see https://github.com/staltz/sodium-native-nodejs-mobile but I'm hitting a runtime error:

Error: dlopen failed: library "libsodium.so" not found
at Object.Module._extensions..node (internal/modules/cjs/loader.js:717:18)
at Module.load (internal/modules/cjs/loader.js:598:32)
at tryModuleLoad (internal/modules/cjs/loader.js:537:12)
at Function.Module._load (internal/modules/cjs/loader.js:529:3)
at Module.require (internal/modules/cjs/loader.js:636:17)
at require (internal/modules/cjs/helpers.js:20:18)
at /data/data/se.manyver/files/nodejs-project/index.js:16:29
at bindings (/data/data/se.manyver/files/nodejs-project/index.js:69758:44)
at Object.84wOzamum2jP+/5fIqg6RNZF+/nib1rh845LX3EpSg8= (/data/data/se.manyver/files/nodejs-project/index.js:19951:33)

I have tried to put libsodium.so in node_modules/sodium-native-nodejs-mobile/build/Release but that has had no effect (with or without libsodium.so in that directory, I get the same runtime error).

I'm running out of ideas, it could either be that libsodium.so should have been included inside build/Release/sodium.node somehow, or it could be that I'm putting this file in the wrong directory. Any suggestions are welcome...

[jaimecbernardo]

Hi @staltz ,

I'm not aware about the internals of sodium, but one way of trying to debug what ends up on the apk would be to rename it to a zip file and looking into its contents (the .apk is essentially a Zip file).

Try to follow the require logic on your code. For example, what is your index.js requiring?

[staltz]

So, regarding the compilation of the *.node and the *.so, everything is working (see https://github.com/sodium-friends/sodium-native/issues/111), what's not working is the handling of the *.so file, that is: (1) moving it to the correct folder post-compilation, (2) runtime detection and loading of *.so.

If I copy-paste libsodium.so into android/app/src/main/jniLibs, everything works, because React Native's SoLoader will take care of detecting it and loading it. But if I don't do this manual step, it doesn't work.

One thing I noticed is that this node module (sodium-native-nodejs-mobile) puts libsodium.so in the folder build/Release, but android/build/nodejs-native-assets/nodejs-native-assets-armeabi-v7a/node_modules/sodium-native-nodejs-mobile/build/Release is missing libsodium.so. So here's my question: does nodejs-mobile delete all files except *.node? If that's the case then it doesn't support dynamic loading when the node module ships the *.so too.

[staltz]

And regarding the structure of the apk,

.
├── AndroidManifest.xml
├── assets
│   ├── builtin_modules
│   ├── dir.list
│   ├── file.list
│   ├── fonts
│   ├── index.android.bundle
│   ├── nodejs-native-assets-arm64-v8a
│   ├── nodejs-native-assets-armeabi-v7a
│   │   ├── dir.list
│   │   ├── file.list 
│   │   └── node_modules
│   │       └── sodium-native-nodejs-mobile
│   │           └── build
│   │               └── Release
│   │                   ├── sodium.node
│   │                   └── libsodium.so  <-------- IT SHOULD END UP HERE
│   └── nodejs-project
├── classes2.dex
├── classes.dex
├── lib
│   ├── arm64-v8a
│   └── armeabi-v7a
│       ├── libc++_shared.so
│       ├── ...
│       └── libsodium.so              <---------- ALTERNATIVELY, IT SHOULD END UP HERE
├── META-INF
│   └── ...
├── okhttp3
│   └── ...
├── res
│   └── ...
└── resources.arsc

But in reality it ends up nowhere because it seems to be deleted as a sibling of sodium.node.

[jaimecbernardo]

Hi @staltz ,

The issue is that only the .node files are being copied to the native assets folder:
https://github.com/JaneaSystems/nodejs-mobile-react-native/blob/cf4c90984e010176f85e76c7931ca8e58e229e57/android/build.gradle#L390

Try to include .so files in there too.
By the way, what about inside the APK nodejs-project path? It's possible that the Android Packager is treating .so files differently too.

[staltz]

@jaimecbernardo Thanks for the comment

If I put the .so files in android/build/nodejs-native-assets they correctly end up in the APK's assets/nodejs-native-assets-____ as a sibling of .node files. But in runtime it still crashes with

E NODEJS-MOBILE: WARNING: linker: /data/data/se.manyver/files/nodejs-project/node_modules/sodium-native-nodejs-mobile/build/Release/sodium.node: unused DT entry: type 0x1d arg 0x5fcc
E NODEJS-MOBILE: Error: dlopen failed: library "libsodium.so" not found

While the ELF header for sodium.node is:

Dynamic section at offset 0x376e8 contains 35 entries:
  Tag        Type                         Name/Value
 0x00000003 (PLTGOT)                     0x38a7c
 0x00000002 (PLTRELSZ)                   2800 (bytes)
 0x00000017 (JMPREL)                     0xa670
 0x00000014 (PLTREL)                     REL
 0x00000011 (REL)                        0x9f08
 0x00000012 (RELSZ)                      1896 (bytes)
 0x00000013 (RELENT)                     8 (bytes)
 0x6ffffffa (RELCOUNT)                   6
 0x00000006 (SYMTAB)                     0x1cc
 0x0000000b (SYMENT)                     16 (bytes)
 0x00000005 (STRTAB)                     0x228c
 0x0000000a (STRSZ)                      24532 (bytes)
 0x6ffffef5 (GNU_HASH)                   0x8260
 0x00000004 (HASH)                       0x8a58
 0x00000001 (NEEDED)                     Shared library: [libsodium.so]   <--------------
 0x00000001 (NEEDED)                     Shared library: [liblog.so]
 0x00000001 (NEEDED)                     Shared library: [libnode.so]
 0x00000001 (NEEDED)                     Shared library: [libc++_shared.so]
 0x00000001 (NEEDED)                     Shared library: [libm.so]
 0x00000001 (NEEDED)                     Shared library: [libdl.so]
 0x00000001 (NEEDED)                     Shared library: [libc.so]
 0x0000000e (SONAME)                     Library soname: [sodium.node]
 0x0000001a (FINI_ARRAY)                 0x38520
 0x0000001c (FINI_ARRAYSZ)               8 (bytes)
 0x00000019 (INIT_ARRAY)                 0x386e4
 0x0000001b (INIT_ARRAYSZ)               4 (bytes)
 0x0000001d (RUNPATH)                    Library runpath: [$ORIGIN]    <-----------------
 0x0000001e (FLAGS)                      BIND_NOW
 0x6ffffffb (FLAGS_1)                    Flags: NOW
 0x6ffffff0 (VERSYM)                     0x9ab4
 0x6ffffffc (VERDEF)                     0x9ecc
 0x6ffffffd (VERDEFNUM)                  1
 0x6ffffffe (VERNEED)                    0x9ee8
 0x6fffffff (VERNEEDNUM)                 1
 0x00000000 (NULL)                       0x0
``

[jaimecbernardo]

Hi @staltz,

It's possible that android/app/src/main/jniLibs is a necessity, since they'll end up in a different place in the APK, then.

Something similar to:

task "CopySodiumSo${abi_name}" (type:Copy) {
    dependsOn "BuildNpmModules${abi_name}"
    description = "Copying libsodium.so for ${abi_name}."
    from "${rootProject.buildDir}/nodejs-native-assets-temp-build/nodejs-native-assets-${abi_name}/nodejs-project/node_modules/sodium-native-nodejs-mobile/build/Release/"
    into "${rootProject.projectDir}/app/src/main/jniLibs/${abi_name}/"
    include 'libsodium.so'
}

tasks.getByPath(":${project.name}:preBuild").dependsOn "CopySodiumSo${abi_name}"        

Add it somewhere around here, I'd say.

[staltz]

Thanks @jaimecbernardo, since that script would have been anyway specific to my app and the library sodium-native-nodejs-mobile, I ended up writing that copy script in a different way and in bash, but the end result is the same, and yes, it now works.

I think I'm fine with this, it's not the best but it works, but I think the question that remains is how should node modules that provide dynamic libraries be included by nodejs-mobile, in general, and in an automated way.

For instance, maybe nodejs-mobile's gradle script can run readelf of each .node, detect unusual .so files they need, and search for those files, and place them in jniLibs?

[staltz]

I started working on sodium-native supporting iOS, see this repo: https://github.com/staltz/sodium-native-nodejs-mobile

I got quite far in the process of compiling sodium-native (and its subdep libsodium) to iOS arm64, but what I'm thinking now is what to do with this libsodium.so file that gets moved around. In the case of iOS, will the nodejs-mobile tooling convert the .so to a .framework?

[calm-rad]

Hi @staltz I'm attempting to get sodium-native working on nodejs-mobile too and I'm running into a C compiler cannot create executables error when I try to run an iOS build off my app on xcode using your sodium-native-nodejs-mobile library.

After digging around I think it's some issue with the ios.sh in the libsodium module that tries to execute in preinstall.js. Unfortunately that's the farthest I've gotten and I'm very new to this stuff.

Did you encounter this when you were getting the library to work? And do you have any suggestions or potential solutions?

I wasn't sure if it was more appropriate to comment here or open an issue on your repo so apologies if this is the wrong place.

UPDATE:

I attempted a to build on Android and I seem to be getting a similar error there as well. Here's the output:

> [email protected] install /Users/User/Documents/App/android/build/nodejs-native-assets-temp-build/nodejs-native-assets-x86/nodejs-project/node_modules/sodium-native
> node-gyp-build "node preinstall.js" "node postinstall.js"

gyp info it worked if it ends with ok
gyp verb cli [ '/usr/local/bin/node',
gyp verb cli   '/Users/User/Documents/App/node_modules/nodejs-mobile-gyp/bin/node-gyp.js',
gyp verb cli   'rebuild' ]
gyp info using [email protected]
gyp info using [email protected] | darwin | x64
gyp verb command rebuild []
gyp verb command clean []
gyp verb clean removing "build" directory
gyp verb command configure []
gyp verb check python checking for Python executable "python2" in the PATH
gyp verb `which` succeeded python2 /usr/bin/python2
gyp verb check python version `/usr/bin/python2 -c "import sys; print "2.7.16
gyp verb check python version .%s.%s" % sys.version_info[:3];"` returned: %j
gyp verb get node dir compiling against specified --nodedir dev files: /Users/User/Documents/App/node_modules/nodejs-mobile-react-native/android/libnode/
gyp verb build dir attempting to create "build" dir: /Users/User/Documents/App/android/build/nodejs-native-assets-temp-build/nodejs-native-assets-x86/nodejs-project/node_modules/sodium-native/build
gyp verb build dir "build" dir needed to be created? /Users/User/Documents/App/android/build/nodejs-native-assets-temp-build/nodejs-native-assets-x86/nodejs-project/node_modules/sodium-native/build
gyp verb build/config.gypi creating config file
gyp verb build/config.gypi writing out config file: /Users/User/Documents/App/android/build/nodejs-native-assets-temp-build/nodejs-native-assets-x86/nodejs-project/node_modules/sodium-native/build/config.gypi
gyp verb config.gypi checking for gypi file: /Users/User/Documents/App/android/build/nodejs-native-assets-temp-build/nodejs-native-assets-x86/nodejs-project/node_modules/sodium-native/config.gypi
gyp verb common.gypi checking for gypi file: /Users/User/Documents/App/android/build/nodejs-native-assets-temp-build/nodejs-native-assets-x86/nodejs-project/node_modules/sodium-native/common.gypi
gyp verb gyp gyp format was not specified; forcing "make"
gyp info spawn /usr/bin/python2
gyp info spawn args [ '/Users/User/Documents/App/node_modules/nodejs-mobile-gyp/gyp/gyp_main.py',
gyp info spawn args   'binding.gyp',
gyp info spawn args   '-f',
gyp info spawn args   'make-android',
gyp info spawn args   '-I',
gyp info spawn args   '/Users/User/Documents/App/android/build/nodejs-native-assets-temp-build/nodejs-native-assets-x86/nodejs-project/node_modules/sodium-native/build/config.gypi',
gyp info spawn args   '-I',
gyp info spawn args   '/Users/User/Documents/App/node_modules/nodejs-mobile-gyp/addon.gypi',
gyp info spawn args   '-I',
gyp info spawn args   '/Users/User/Documents/App/node_modules/nodejs-mobile-react-native/android/libnode/include/node/common.gypi',
gyp info spawn args   '-Dlibrary=shared_library',
gyp info spawn args   '-Dvisibility=default',
gyp info spawn args   '-Dnode_root_dir=/Users/User/Documents/App/node_modules/nodejs-mobile-react-native/android/libnode/',
gyp info spawn args   '-Dnode_gyp_dir=/Users/User/Documents/App/node_modules/nodejs-mobile-gyp',
gyp info spawn args   '-Dnode_lib_file=/Users/User/Documents/App/node_modules/nodejs-mobile-react-native/android/libnode/$(Configuration)/node.lib',
gyp info spawn args   '-Dmodule_root_dir=/Users/User/Documents/App/android/build/nodejs-native-assets-temp-build/nodejs-native-assets-x86/nodejs-project/node_modules/sodium-native',
gyp info spawn args   '-Dnode_engine=v8',
gyp info spawn args   '--depth=.',
gyp info spawn args   '--no-parallel',
gyp info spawn args   '--generator-output',
gyp info spawn args   'build',
gyp info spawn args   '-Goutput_dir=.' ]
gyp verb command build []
gyp verb build type Release
gyp verb architecture x86
gyp verb node dev dir /Users/User/Documents/App/node_modules/nodejs-mobile-react-native/android/libnode/
gyp verb `which` succeeded for `make` /usr/bin/make
gyp info spawn make
gyp info spawn args [ 'V=1', 'BUILDTYPE=Release', '-C', 'build' ]
make: *** No rule to make target `/Users/User/Documents/App/android/build/nodejs-native-assets-temp-build/nodejs-native-assets-x86/nodejs-project/node_modules/sodium-native/lib/android-x86/libsodium.so', needed by `Release/libsodium.so'.  Stop.
gyp ERR! build error 
gyp ERR! stack Error: `make` failed with exit code: 2
gyp ERR! stack     at ChildProcess.onExit (/Users/User/Documents/App/node_modules/nodejs-mobile-gyp/lib/build.js:248:23)
gyp ERR! stack     at ChildProcess.emit (events.js:182:13)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:240:12)
gyp ERR! System Darwin 19.2.0
gyp ERR! command "/usr/local/bin/node" "/Users/User/Documents/App/node_modules/nodejs-mobile-gyp/bin/node-gyp.js" "rebuild"
gyp ERR! cwd /Users/User/Documents/App/android/build/nodejs-native-assets-temp-build/nodejs-native-assets-x86/nodejs-project/node_modules/sodium-native
gyp ERR! node -v v10.15.0
gyp ERR! node-gyp -v v0.3.1
gyp ERR! not ok 
npm verb lifecycle [email protected]~install: unsafe-perm in lifecycle true
npm verb lifecycle [email protected]~install: PATH: /usr/local/lib/node_modules/npm/node_modules/npm-lifecycle/node-gyp-bin:/Users/User/Documents/App/android/build/nodejs-native-assets-temp-build/nodejs-native-assets-x86/nodejs-project/node_modules/sodium-native/node_modules/.bin:/Users/User/Documents/App/android/build/nodejs-native-assets-temp-build/nodejs-native-assets-x86/nodejs-project/node_modules/.bin:/Users/User/Documents/App/android/../nodejs-assets/nodejs-project/node_modules/.bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin
npm verb lifecycle [email protected]~install: CWD: /Users/User/Documents/App/android/build/nodejs-native-assets-temp-build/nodejs-native-assets-x86/nodejs-project/node_modules/sodium-native
npm info lifecycle [email protected]~install: Failed to exec install script
npm verb stack Error: [email protected] install: `node-gyp-build "node preinstall.js" "node postinstall.js"`
npm verb stack Exit status 1
npm verb stack     at EventEmitter.<anonymous> (/usr/local/lib/node_modules/npm/node_modules/npm-lifecycle/index.js:301:16)
npm verb stack     at EventEmitter.emit (events.js:182:13)
npm verb stack     at ChildProcess.<anonymous> (/usr/local/lib/node_modules/npm/node_modules/npm-lifecycle/lib/spawn.js:55:14)
npm verb stack     at ChildProcess.emit (events.js:182:13)
npm verb stack     at maybeClose (internal/child_process.js:962:16)
npm verb stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:251:5)
npm verb pkgid [email protected]
npm verb cwd /Users/User/Documents/App/android/build/nodejs-native-assets-temp-build/nodejs-native-assets-x86/nodejs-project
npm verb Darwin 19.2.0
npm verb argv "/usr/local/bin/node" "/usr/local/bin/npm" "--verbose" "rebuild" "--build-from-source"
npm verb node v10.15.0
npm verb npm  v6.4.1
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] install: `node-gyp-build "node preinstall.js" "node postinstall.js"`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the [email protected] install script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
npm verb exit [ 1, true ]
npm timing npm Completed in 7842ms

npm ERR! A complete log of this run can be found in:
npm ERR!     /Users/User/.npm/_logs/2020-01-09T18_52_58_231Z-debug.log

> Task :nodejs-mobile-react-native:BuildNpmModulesx86 FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':nodejs-mobile-react-native:BuildNpmModulesx86'.