Document and fail fast if tmp directory is noexec #252

melissalinkert · 2024-06-03T21:38:14Z

Updates readme to reference the known noexec issue, and tries to detect and fail conversion quickly if that is the case. The check is very heavy-handed, as it should always fail if java.io.tmpdir is noexec - independent of the input format or conversion options. We could make that more lenient if any strong opinions.

sbesson

In a system where /tmp partition is mounted with noexec options i.e.

omero@apps:/tmp$ mount | grep /tmp
/dev/sda on /tmp type ext4 (rw,noexec,relatime,discard)

running bioformats2raw e.g. against a NPDI file will fail with an exception looking like the following:

omero@apps:/data$ ~/apps/bioformats2raw-0.9.2/bin/bioformats2raw CMU-1.ndpi CMU-1.zarr
2024-06-28 08:50:16,044 [main] WARN  c.g.bioformats2raw.OpenCVTools - Could not load OpenCV libraries
java.lang.UnsatisfiedLinkError: /tmp/opencv_openpnp17542193582129005746/nu/pattern/opencv/linux/x86_64/libopencv_java470.so: /tmp/opencv_openpnp17542193582129005746/nu/pattern/opencv/linux/x86_64/libopencv_java470.so: failed to map segment from shared object
	at java.base/java.lang.ClassLoader$NativeLibrary.load0(Native Method)
	at java.base/java.lang.ClassLoader$NativeLibrary.load(ClassLoader.java:2450)
	at java.base/java.lang.ClassLoader$NativeLibrary.loadLibrary(ClassLoader.java:2506)
	at java.base/java.lang.ClassLoader.loadLibrary0(ClassLoader.java:2705)
	at java.base/java.lang.ClassLoader.loadLibrary(ClassLoader.java:2635)
	at java.base/java.lang.Runtime.load0(Runtime.java:768)
	at java.base/java.lang.System.load(System.java:1850)
	at nu.pattern.OpenCV$LocalLoader.<init>(OpenCV.java:330)
	at nu.pattern.OpenCV$LocalLoader.<init>(OpenCV.java:326)
	at nu.pattern.OpenCV$LocalLoader$Holder.<clinit>(OpenCV.java:336)
	at nu.pattern.OpenCV$LocalLoader.getInstance(OpenCV.java:340)
	at nu.pattern.OpenCV.loadLocally(OpenCV.java:323)
	at com.glencoesoftware.bioformats2raw.OpenCVTools.loadOpenCV(OpenCVTools.java:34)
	at com.glencoesoftware.bioformats2raw.Converter.call(Converter.java:1151)
	at com.glencoesoftware.bioformats2raw.Converter.call(Converter.java:104)
	at picocli.CommandLine.executeUserObject(CommandLine.java:2041)
	at picocli.CommandLine.access$1500(CommandLine.java:148)
	at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2461)
	at picocli.CommandLine$RunLast.handle(CommandLine.java:2453)
	at picocli.CommandLine$RunLast.handle(CommandLine.java:2415)
	at picocli.CommandLine$AbstractParseResultHandler.handleParseResult(CommandLine.java:2264)
	at picocli.CommandLine.parseWithHandlers(CommandLine.java:2664)
	at picocli.CommandLine.parseWithHandler(CommandLine.java:2599)
	at picocli.CommandLine.call(CommandLine.java:2875)
	at com.glencoesoftware.bioformats2raw.Converter.main(Converter.java:2974)
2024-06-28 08:50:16,053 [main] WARN  c.g.bioformats2raw.OpenCVTools - Could not load native library opencv_java470
java.lang.UnsatisfiedLinkError: no opencv_java470 in java.library.path: [/usr/java/packages/lib, /usr/lib/x86_64-linux-gnu/jni, /lib/x86_64-linux-gnu, /usr/lib/x86_64-linux-gnu, /usr/lib/jni, /lib, /usr/lib]
	at java.base/java.lang.ClassLoader.loadLibrary(ClassLoader.java:2678)
	at java.base/java.lang.Runtime.loadLibrary0(Runtime.java:830)
	at java.base/java.lang.System.loadLibrary(System.java:1886)
	at com.glencoesoftware.bioformats2raw.OpenCVTools.loadOpenCV(OpenCVTools.java:41)
	at com.glencoesoftware.bioformats2raw.Converter.call(Converter.java:1151)
	at com.glencoesoftware.bioformats2raw.Converter.call(Converter.java:104)
	at picocli.CommandLine.executeUserObject(CommandLine.java:2041)
	at picocli.CommandLine.access$1500(CommandLine.java:148)
	at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2461)
	at picocli.CommandLine$RunLast.handle(CommandLine.java:2453)
	at picocli.CommandLine$RunLast.handle(CommandLine.java:2415)
	at picocli.CommandLine$AbstractParseResultHandler.handleParseResult(CommandLine.java:2264)
	at picocli.CommandLine.parseWithHandlers(CommandLine.java:2664)
	at picocli.CommandLine.parseWithHandler(CommandLine.java:2599)
	at picocli.CommandLine.call(CommandLine.java:2875)
	at com.glencoesoftware.bioformats2raw.Converter.main(Converter.java:2974)
Exception in thread "main" picocli.CommandLine$ExecutionException: Error while calling command (com.glencoesoftware.bioformats2raw.Converter@68e5eea7): java.io.IOException: JPEG service failed to load Turbo JPEG library
	at picocli.CommandLine.executeUserObject(CommandLine.java:2050)
	at picocli.CommandLine.access$1500(CommandLine.java:148)
	at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2461)
	at picocli.CommandLine$RunLast.handle(CommandLine.java:2453)
	at picocli.CommandLine$RunLast.handle(CommandLine.java:2415)
	at picocli.CommandLine$AbstractParseResultHandler.handleParseResult(CommandLine.java:2264)
	at picocli.CommandLine.parseWithHandlers(CommandLine.java:2664)
	at picocli.CommandLine.parseWithHandler(CommandLine.java:2599)
	at picocli.CommandLine.call(CommandLine.java:2875)
	at com.glencoesoftware.bioformats2raw.Converter.main(Converter.java:2974)
Caused by: java.io.IOException: JPEG service failed to load Turbo JPEG library
	at loci.formats.in.NDPIReader.initFile(NDPIReader.java:332)
	at loci.formats.FormatReader.setId(FormatReader.java:1480)
	at loci.formats.ImageReader.setId(ImageReader.java:865)
	at com.glencoesoftware.bioformats2raw.Converter.getBaseReaderClass(Converter.java:2791)
	at com.glencoesoftware.bioformats2raw.Converter.convert(Converter.java:1228)
	at com.glencoesoftware.bioformats2raw.Converter.call(Converter.java:1204)
	at com.glencoesoftware.bioformats2raw.Converter.call(Converter.java:104)
	at picocli.CommandLine.executeUserObject(CommandLine.java:2041)
	... 9 more

With this change the same command results in the following error

omero@apps:/data$ ~/apps/bioformats2raw-0.10.0-SNAPSHOT/bin/bioformats2raw CMU-1.ndpi CMU-1.zarr
Exception in thread "main" picocli.CommandLine$ExecutionException: Error while calling command (com.glencoesoftware.bioformats2raw.Converter@68e5eea7): java.lang.RuntimeException: /tmp is noexec; fix it or specify a different java.io.tmpdir
	at picocli.CommandLine.executeUserObject(CommandLine.java:2050)
	at picocli.CommandLine.access$1500(CommandLine.java:148)
	at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2461)
	at picocli.CommandLine$RunLast.handle(CommandLine.java:2453)
	at picocli.CommandLine$RunLast.handle(CommandLine.java:2415)
	at picocli.CommandLine$AbstractParseResultHandler.handleParseResult(CommandLine.java:2264)
	at picocli.CommandLine.parseWithHandlers(CommandLine.java:2664)
	at picocli.CommandLine.parseWithHandler(CommandLine.java:2599)
	at picocli.CommandLine.call(CommandLine.java:2875)
	at com.glencoesoftware.bioformats2raw.Converter.main(Converter.java:3017)
Caused by: java.lang.RuntimeException: /tmp is noexec; fix it or specify a different java.io.tmpdir
	at com.glencoesoftware.bioformats2raw.Converter.call(Converter.java:1190)
	at com.glencoesoftware.bioformats2raw.Converter.call(Converter.java:104)
	at picocli.CommandLine.executeUserObject(CommandLine.java:2041)
	... 9 more

Following the recommendations

omero@apps:/data$ JAVA_OPTS="-Djava.io.tmpdir=/opt/omero/tmp" ~/apps/bioformats2raw-0.10.0-SNAPSHOT/bin/bioformats2raw CMU-1.ndpi CMU-1.zarr
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.esotericsoftware.reflectasm.AccessClassLoader (file:/opt/omero/apps/bioformats2raw-0.10.0-SNAPSHOT/lib/reflectasm-1.11.9.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain)
WARNING: Please consider reporting this to the maintainers of com.esotericsoftware.reflectasm.AccessClassLoader
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
omero@apps:/data$ ls CMU-1.zarr/
0  1  OME

The requirement to mount /tmp as noexec is a guideline from the CIS security benchmarks. This means we should expect increasingly more systems to start enforcing this recommendation.
These mount options are at odds with the current logic of the native-lib-loader library used by Bio-Formats which extracts native libraries bundled in a JAR under java.io.tmpdir, usually /tmp, with executable permissions. This exact issue has already been reported in https://forum.image.sc/t/82646 and https://forum.image.sc/t/81868 and will affect file formats using libraries like libjpegturbo or libjxr.

Discussing potential upstream resolutions with @melissalinkert, one of the problem is that there is no authoritative way to identify an alternative temporary directory with write & executable permissions in all scenarios. As discussed recently with @stick, an application like OMERO.server might be able to create its own internal temporary directory like var/tmp and set java.io.tmpdir. On the other side, a library bioformats2raw might be used in variety of environments including containers, HPC.

Functionally, this PR is definitely an improvement, documents a known issue and provides the end-user with a more informative error on how to resolve it. As indicated in the description, the biggest concern is that the proposed implementation will always fail if /tmp is mounted as noexec even if no native library is required. It would be useful to get additional feedback from people working directly with end-users to report on the potential negative effect before spending additional time in investigating more lenient ways to send the error message.

joshmoore · 2024-07-02T01:26:21Z

Two quick thoughts:

if possible, making this a flag might be worth the visibility and usability if this is going to become a more frequent issue
(possibly for upstream) I could also see this as a (personal) cache-dir scenario rather than temp-dir. If so, one could conceivably keep the files around until the next upgrade to speed up start up.

melissalinkert · 2024-07-12T20:42:18Z

As discussed earlier today, the next steps here are:

keep the exception by default, but add an option to warn instead
make the readme changes a separate section (so we can link to it), and explain the issue a little more
delete the "check" file right away instead of on exit
include in a 0.10.0 (minor, not patch) release and make sure release notes flag this as a potentially breaking change

…xception

melissalinkert · 2024-07-15T21:50:20Z

keep the exception by default, but add an option to warn instead

make the readme changes a separate section (so we can link to it), and explain the issue a little more

delete the "check" file right away instead of on exit

include in a 0.10.0 (minor, not patch) release and make sure release notes flag this as a potentially breaking change

All except the actual release should be covered by 9dca8a9, cc2ec93, and 862e4ac.

sbesson

Tested in a system with /tmp mounted as noexec.

bioformats2raw test.fake test.zarr fails but bioformats2raw test.fake test.zarr --warn-no-exec executes the conversion with a warning as expected.

Only issue that arose from the testing is the cleanup of the temporary test files when the command fails with the exception

sbesson · 2024-07-18T12:11:16Z

src/main/java/com/glencoesoftware/bioformats2raw/Converter.java

+        throw new RuntimeException(msg);
+      }
+    }
+    tmpdirCheck.delete();


In the scenario where /tmp is mounted as noxec and --warn-no-exec is not passed to the command, this deletion will never be executed as per the throw call a few lines earlier. Would it make sense to wrap this in a try/finally block to ensure temporary test files are always cleaned up?

README.md

Co-authored-by: Sébastien Besson <[email protected]>

sbesson

Retested with the last commit and the temporary file created for checking the executable state of java.io.tmpdir is now cleaned up on every command invocation independently of whether --warn-no-exec is passed.

chris-allan · 2024-07-19T10:32:13Z

src/main/java/com/glencoesoftware/bioformats2raw/Converter.java

+      boolean success = tmpdirCheck.setExecutable(true);
+      if (!success || !tmpdirCheck.canExecute()) {
+        String msg = System.getProperty("java.io.tmpdir") +
+          " is noexec; fix it or specify a different java.io.tmpdir";


Link to https://github.com/glencoesoftware/bioformats2raw/blob/master/README.md#temporary-directory-usage?

See c772b46.

stick

All fine with the behavior and documentation from my end.

Document and fail fast if tmp directory is noexec

57f8bbe

melissalinkert requested review from stick, sbesson, mabruce and erindiel June 3, 2024 21:38

sbesson reviewed Jun 28, 2024

View reviewed changes

melissalinkert added 3 commits July 15, 2024 14:19

Delete tmp check file right away

9dca8a9

Add --warn-no-exec option to log a warning instead of throwing an e…

cc2ec93

…xception

Split tmp directory notes into their own section

862e4ac

melissalinkert requested review from sbesson and chris-allan July 15, 2024 21:50

melissalinkert added this to the 0.10.0 milestone Jul 15, 2024

sbesson requested changes Jul 18, 2024

View reviewed changes

melissalinkert and others added 2 commits July 18, 2024 08:52

Fix JAVA_OPTS syntax

80f3ffb

Co-authored-by: Sébastien Besson <[email protected]>

One more try at deleting the tmp check file correctly

44d031f

melissalinkert requested a review from sbesson July 18, 2024 14:51

sbesson approved these changes Jul 19, 2024

View reviewed changes

chris-allan reviewed Jul 19, 2024

View reviewed changes

Link to README in tmp directory message

c772b46

stick approved these changes Jul 22, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Document and fail fast if tmp directory is noexec #252

Document and fail fast if tmp directory is noexec #252

melissalinkert commented Jun 3, 2024

sbesson left a comment

joshmoore commented Jul 2, 2024

melissalinkert commented Jul 12, 2024

melissalinkert commented Jul 15, 2024

sbesson left a comment

sbesson Jul 18, 2024

sbesson left a comment

chris-allan Jul 19, 2024

melissalinkert Jul 19, 2024

stick left a comment

Document and fail fast if tmp directory is noexec #252

Are you sure you want to change the base?

Document and fail fast if tmp directory is noexec #252

Conversation

melissalinkert commented Jun 3, 2024

sbesson left a comment

Choose a reason for hiding this comment

joshmoore commented Jul 2, 2024

melissalinkert commented Jul 12, 2024

melissalinkert commented Jul 15, 2024

sbesson left a comment

Choose a reason for hiding this comment

sbesson Jul 18, 2024

Choose a reason for hiding this comment

sbesson left a comment

Choose a reason for hiding this comment

chris-allan Jul 19, 2024

Choose a reason for hiding this comment

melissalinkert Jul 19, 2024

Choose a reason for hiding this comment

stick left a comment

Choose a reason for hiding this comment