Add ability for static search locations

[tresf]

Summary

The native-lib-loader should allow specifying a library in a static location for edge-case scenarios, usually imposed by security restrictions, constraints or sandboxing.

Edit: Static locations may also offer slight performance benefits as well, eliminating the need for unzipping (CPU/disk) activity, see discussion here.

Details

Technically, native-lib-loader allows for static search locations, but it requires implementing the JniExtractor class or overriding the behavior of the DefaultJniExractor class, as -- by design -- they both intend to perform an extract operation prior to loading a native library. This extraction operation doesn't work in all environments.

I'll quote a sister project -- JNA -- documentation, I think it words this problem well and explains as to why this can be needed...

• When [...] classes are loaded, the native shared library [...] is loaded as well. An attempt is made to load it from the any paths defined in jna.boot.library.path (if defined), then the system library path using System.loadLibrary(java.lang.String), unless jna.nosys=true.
• If not found, the appropriate library will be extracted from the class path (into a temporary directory if found within a jar file) and loaded from there, unless jna.noclasspath=true.
• If your system has additional security constraints regarding execution or load of files (SELinux, for example), you should probably install the native library in an accessible location and configure your system accordingly, rather than relying on JNA to extract the library from its own jar file.

The last bullet is the point I'd like to focus on since this same use-case exists for native-lib-loader. For example, if a native library is intended to be distributed with a Java application that's distributed from the Apple AppStore, sandboxing is a hard-requirement, and like the aforementioned SELinux use-case, sandboxing can prevent loading a library from arbitrary locations (such as $TEMP), quoting an Apple employee on the Apple Developer forums:

I think you’ll run into problems there. Specifically, modern versions of the system prevent an app from referencing libraries outside of the app (other than system libraries). See Gatekeeper Changes in OS X v10.10.4 and Later in Technote 2206 OS X Code Signing In Depth

Furthermore, the JNA portion "unless jna.nosys=true", I find to be increasingly important as client environments may have identically named libraries in search paths that the client has little or no control over. This is a separate issue, but may be tackled as part of the same enhancement.

Workaround

Providing a stub extractor can handle this issue:

Click to see DefaultJniExtractorStub.java

/**
 * License: https://opensource.org/licenses/BSD-3-Clause
 */
package jssc;

import org.scijava.nativelib.DefaultJniExtractor;
import org.scijava.nativelib.NativeLibraryUtil;

import java.io.File;
import java.io.IOException;

/**
 * @author A. Tres Finocchiaro
 *
 * Stub <code>DefaultJniExtractor</code> class to allow native-lib-loader to conditionally
 * use a statically defined native search path <code>bootPath</code> when provided.
 */
public class DefaultJniExtractorStub extends DefaultJniExtractor {
    private File bootPath;
    private boolean useStub;

    /**
     * Default constructor
     */
    public DefaultJniExtractorStub(Class libraryJarClass) throws IOException {
        super(libraryJarClass);
        useStub = false;
    }

    /**
     * Force native-lib-loader to first look in the location defined as <code>bootPath</code>
     * prior to extracting a native library, useful for sandboxed environments.
     *  <code>
     *  NativeLoader.setJniExtractor(new DefaultJniExtractorStub(null, "/opt/nativelibs")));
     *  NativeLoader.loadLibrary("mylibrary");
     *  </code>
     */
    public DefaultJniExtractorStub(Class libraryJarClass, String bootPath) throws IOException {
        this(libraryJarClass);
        this.bootPath = new File(bootPath);

        if(bootPath != null) {
            File bootTest = new File(bootPath);
            if(bootTest.exists()) {
                // assume a static, existing directory will contain the native libs
                this.useStub = true;
            } else {
                System.err.println("WARNING " + DefaultJniExtractorStub.class.getCanonicalName() + ": Boot path " + bootPath + " not found, falling back to default extraction behavior.");
            }
        }
    }

    /**
     * If a <code>bootPath</code> was provided to the constructor and exists,
     * calculate the <code>File</code> path without any extraction logic.
     *
     * If a <code>bootPath</code> was NOT provided or does NOT exist, fallback on
     * the default extraction behavior.
     */
    @Override
    public File extractJni(String libPath, String libName) throws IOException {
        // Lie and pretend it's already extracted at the bootPath location
        if(useStub) {
            return new File(bootPath, NativeLibraryUtil.getPlatformLibraryName(libName));
        }
        // Fallback on default behavior
        return super.extractJni(libPath, libName);
    }

    @Override
    public void extractRegistered() throws IOException {
        if(useStub) {
            return; // no-op
        }
        super.extractRegistered();
    }
}

Usage:

+ NativeLoader.setJniExtractor(new DefaultJniExtractorStub(null, "/opt/libs"));
  NativeLoader.loadLibrary("mylibrary");

Caveats

Due to the library loading order in native-lib-loader, any System locations will be always be preferred, which can cause compatibility issues if the system was modified (probably warrants a separate bug report).

native-lib-loader/src/main/java/org/scijava/nativelib/NativeLoader.java

Lines 134 to 142 in 56cdf62

	try {
	// try to load library from classpath
	System.loadLibrary(libName);
	}
	catch (final UnsatisfiedLinkError e) {
	if (NativeLibraryUtil.loadNativeLibrary(jniExtractor, libName,
	searchPaths)) return;
	throw new IOException("Couldn't load library library " + libName, e);
	}

[ctrueden]

Thanks for the detailed writeup, @tresf!

About the eager System.loadLibrary call: a backwards-compatible way to address this would be to make a new method:

public static void loadLibrary(final String libName, final boolean trySystemFirst,
		final String... searchPaths) throws IOException

which lets the caller control whether to do a System.loadLibrary before falling back. And then the existing loadLibrary signature can pass trySystemFirst=true so that existing code keeps working. What do you think?

[tresf]

What do you think?

First I'd like to preface that often people don't exactly have control over the code that's executing. They might be hitting a bug on only one system and providing a code-less workaround can empower them to manipulate the environment to fix the issue without editing code (e.g. systems administrator or support tech trying to troubleshoot an app crashing).

So I think it's a saner default behavior to mimic JNA's ordering (which native-lib-loader already does, mostly)

• Manual specified location (Missing)
• System location (Matches JNA, minus the conditional)
• Extracted location (Matches JNA, minus the conditional)

.. and match JNA by loading these conditional properties based on a user-editable System property.

So the short solution in my opinion is to simply add a condition like:

String nosys = System.getProperty("native-lib-loader.nosys");
if(nosys == null || !Boolean.parseBoolean(nosys)) {
   // Check system path unless otherwise specified
   System.loadLibrary(libName); 
}

This same condition can be applied to extraction as well, quoting JNA:

To avoid the automatic unpacking (in situations where you want to force a failure if the JNA native library is not properly installed on the system), set the system propertyjna.nounpack=true.

... so to mimic this behavior, something like native-lib-loader.nosys=true (or shorter, nll.nosystem=true) to similarly force the failure (especially useful for unit testing to know your static library or your system library is the one being loaded, even in cases where it's also technically still unzippable).

Sorry to copy so much text and property names from JNA, I'm just rather pleased with how they offer System properties for this, since often I don't want to recompile a dependant library to control the native library structure (e.g. repackaging a third-party lib for compatibility with the AppStore).

This allows something as simple (codeless) as:

_JAVA_OPTIONS="-Dnative-lib-loader.nosys=true"

... or something that can be controlled 100% from within code:

System.setProperty("native-lib-loader.nosys", "true");

It also will theoretically require no API changes to the project.

[tresf]

To add to the above, if the opinion of native-lib-loader is to offer fine-grained control over which libraries load using each technique, a similar convention could be adopted, e.g.:

System.setProperty("native-lib-loader.mylib.nosys", "true");
//                                    ^----- additional namespace for granular control
NativeLoader.loadLibrary("mylib");

... although to that point, I think the last entropy has very few (if any) use-cases and can probably be started as its own request if/when needed.