Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

JDK11 crashed in this base image #37

Open
menghuxiashan opened this issue Dec 24, 2019 · 9 comments
Open

JDK11 crashed in this base image #37

menghuxiashan opened this issue Dec 24, 2019 · 9 comments
Labels
help wanted

Comments

@menghuxiashan
Copy link

menghuxiashan commented Dec 24, 2019 •
edited by frol
Loading

Dockerfile:

FROM frolvlad/alpine-glibc:latest
RUN mkdir -p /usr/java/
ADD jdk-11.0.5_linux-x64_bin.tar.gz /usr/java/
ENV JAVA_HOME=/usr/java/jdk-11.0.5
ENV PATH ${PATH}:${JAVA_HOME}/bin

when enter container and input command:

/ # java -version
java version "11.0.5" 2019-10-15 LTS
Java(TM) SE Runtime Environment 18.9 (build 11.0.5+10-LTS)
Java HotSpot(TM) 64-Bit Server VM 18.9 (build 11.0.5+10-LTS, mixed mode)
[thread 11 also had an error]
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007f28dcb2aab1, pid=6, tid=21
#
# JRE version: Java(TM) SE Runtime Environment (11.0.5+10) (build 11.0.5+10-LTS)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (11.0.5+10-LTS, mixed mode, tiered, compressed oops, g1 gc, linux-amd64)
# Problematic frame:
# C  [libc.so.6+0x11dab1][thread 12 also had an error]

#
# Core dump will be written. Default location: Core dumps may be processed with "/usr/libexec/abrt-hook-ccpp %s %c %p %u %g %t e %P %I %h" (or dumping to //core.6)
#
# An error report file with more information is saved as:
# //hs_err_pid6.log
[thread 9 also had an error]
[thread 16 also had an error]
[thread 13 also had an error]
[thread 7 also had an error]
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
#
Aborted (core dumped)

Why this happened? Can someone help me? Thanks very much!
@frol
Copy link
Member

frol commented Dec 25, 2019

I have no idea why this may happen. I can only suggest using OpenJDK compiled for musl libc.

@xfh
Copy link

xfh commented Mar 2, 2020

FYI: I am maintaining some company internal docker images based on this Dockerfile and the AdoptOpenJDK Version: https://github.com/AdoptOpenJDK/openjdk-docker/blob/master/11/jdk/alpine/Dockerfile.hotspot.releases.full

In the AdoptOpenJDK Dockerfile, ZLIB and GCC_LIBS are installed. I just run a test without ZLIB nor GCC_LIBS and while JDK8 works well, JDK10, 11 and 13 crashed.

@nadavwr
Copy link

nadavwr commented Jun 11, 2020 •
edited
Loading

I just installed Zulu OpenJDK 11.0.7 JRE into this base image:

ADD https://cdn.azul.com/zulu/bin/zulu11.39.15-ca-jre11.0.7-linux_x64.tar.gz /opt/
ENV JAVA_HOME /opt/zulu11.39.15-ca-jre11.0.7-linux_x64

java --version gains me a fairly similar crash:

openjdk 11.0.7 2020-04-14 LTS
OpenJDK Runtime Environment Zulu11.39+15-CA (build 11.0.7+10-LTS)
OpenJDK 64-Bit Server VM Zulu11.39+15-CA (build 11.0.7+10-LTS, mixed mode)
[thread 79 also had an error]#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007fbb5414da41, pid=64, tid=69
#
# JRE version: OpenJDK Runtime Environment (Zulu11.39+15-CA) (11.0.7+10) (build 11.0.7+10-LTS)
# Java VM: OpenJDK 64-Bit Server VM (11.0.7+10-LTS, mixed mode, tiered, compressed oops, g1 gc, linux-amd64)
# Problematic frame:
# C  [libc.so.6+0x11da41]

#
# Core dump will be written. Default location: Core dumps may be processed with "/usr/share/apport/apport %p %s %c %P" (or dumping to //core.64)
#
# An error report file with more information is saved as:
# //hs_err_pid64.log
[thread 70 also had an error][thread 67 also had an error]

[thread 71 also had an error]
[thread 65 also had an error]
#
# If you would like to submit a bug report, please visit:
#   http://www.azulsystems.com/support/
#
Aborted (core dumped)

This is ldd  $JAVA_HOME/bin/java:

	/lib64/ld-linux-x86-64.so.2 (0x7f0ada2d1000)
	libz.so.1 => /lib/libz.so.1 (0x7f0ada0b4000)
	libpthread.so.0 => /lib64/ld-linux-x86-64.so.2 (0x7f0ada2d1000)
	libjli.so => /opt/zulu11.39.15-ca-jre11.0.7-linux_x64/bin/../lib/jli/libjli.so (0x7f0ad9ea4000)
	libdl.so.2 => /lib64/ld-linux-x86-64.so.2 (0x7f0ada2d1000)
	libc.so.6 => /lib64/ld-linux-x86-64.so.2 (0x7f0ada2d1000)
Error relocating /opt/zulu11.39.15-ca-jre11.0.7-linux_x64/bin/../lib/jli/libjli.so: __strdup: symbol not found
Error relocating /opt/zulu11.39.15-ca-jre11.0.7-linux_x64/bin/../lib/jli/libjli.so: __rawmemchr: symbol not found

This is sh /usr/glibc-compat/bin/ldd $JAVA_HOME/bin/java:

	linux-vdso.so.1 (0x00007fff74dde000)
	libz.so.1 => /lib/libz.so.1 (0x00007f6aad6d1000)
	libpthread.so.0 => /usr/glibc-compat/lib/libpthread.so.0 (0x00007f6aad6af000)
	libjli.so => /opt/zulu11.39.15-ca-jre11.0.7-linux_x64/bin/../lib/jli/libjli.so (0x00007f6aad49f000)
	libdl.so.2 => /usr/glibc-compat/lib/libdl.so.2 (0x00007f6aad49a000)
	libc.so.6 => /usr/glibc-compat/lib/libc.so.6 (0x00007f6aad2d5000)
	libc.musl-x86_64.so.1 => /lib/libc.musl-x86_64.so.1 (0x00007f6aad240000)
	/lib64/ld-linux-x86-64.so.2 => /usr/glibc-compat/lib64/ld-linux-x86-64.so.2 (0x00007f6aad8f0000)

I guess java still links against musl instead of glibc. Any ideas?

Setting LD_LIBRARY_PATH to /usr/glibc-compat/lib64:/usr/glibc-compat/lib didn't help.

@nadavwr
Copy link

nadavwr commented Jun 11, 2020 •
edited
Loading

Went over some closed tickets. @frol indicates that ldd output can be misleading here, and suggests prefacing with LD_TRACE_LOADED_OBJECTS=1 instead.

/ # LD_TRACE_LOADED_OBJECTS=1 $JAVA_HOME/bin/java
	linux-vdso.so.1 (0x00007ffe48df2000)
	libz.so.1 => /lib/libz.so.1 (0x00007fe058bf4000)
	libpthread.so.0 => /usr/glibc-compat/lib/libpthread.so.0 (0x00007fe058bd2000)
	libjli.so => /opt/zulu11.39.15-ca-jre11.0.7-linux_x64/bin/../lib/jli/libjli.so (0x00007fe05882b000)
	libdl.so.2 => /usr/glibc-compat/lib/libdl.so.2 (0x00007fe058bcd000)
	libc.so.6 => /usr/glibc-compat/lib/libc.so.6 (0x00007fe058666000)
	libc.musl-x86_64.so.1 => /lib/libc.musl-x86_64.so.1 (0x00007fe058b38000)
	/lib64/ld-linux-x86-64.so.2 (0x00007fe058c10000)

Something is obviously working right. I'm a bit surprised to see libc.musl pop up near the end.
Looks like it is /lib/libz.so.1 that's pulling in libc.musl. Shouldn't I be able to convince libz to link with glibc?

@frol
Copy link
Member

frol commented Jun 11, 2020

Try running the binary explicitly with /usr/glibc-compat/lib64/ld-linux-x86-64.so.2:

/usr/glibc-compat/lib64/ld-linux-x86-64.so.2 $JAVA_HOME/bin/java

@nadavwr
Copy link

nadavwr commented Jun 18, 2020

Same behavior—zlib still leads to musl being loaded:

Ran it with LD_DEBUG=libs to see library loads as they happen. Output for LD_DEBUG=libs /usr/glibc-compat/lib64/ld-linux-x86-64.so.2 $JAVA_HOME/bin/java --version and LD_DEBUG=libs $JAVA_HOME/bin/java --version is identical, so I'm assuming that direct usage of the dynamic linker makes no difference.

# LD_DEBUG=libs /usr/glibc-compat/lib64/ld-linux-x86-64.so.2
$JAVA_HOME/bin/java --version
       214:	find library=libz.so.1 [0]; searching
       214:	 search path=/opt/java/bin/../lib/jli/tls/haswell/x86_64:/opt/java/bin/../lib/jli/tls/haswell:/opt/java/bin/../lib/jli/tls/x86_64:/opt/java/bin/../lib/jli/tls:/opt/java/bin/../lib/jli/haswell/x86_64:/opt/java/bin/../lib/jli/haswell:/opt/java/bin/../lib/jli/x86_64:/opt/java/bin/../lib/jli:/opt/java/bin/../lib/tls/haswell/x86_64:/opt/java/bin/../lib/tls/haswell:/opt/java/bin/../lib/tls/x86_64:/opt/java/bin/../lib/tls:/opt/java/bin/../lib/haswell/x86_64:/opt/java/bin/../lib/haswell:/opt/java/bin/../lib/x86_64:/opt/java/bin/../lib		(RPATH from file /opt/java/bin/java)
       214:	  trying file=/opt/java/bin/../lib/jli/tls/haswell/x86_64/libz.so.1
       214:	  trying file=/opt/java/bin/../lib/jli/tls/haswell/libz.so.1
       214:	  trying file=/opt/java/bin/../lib/jli/tls/x86_64/libz.so.1
       214:	  trying file=/opt/java/bin/../lib/jli/tls/libz.so.1
       214:	  trying file=/opt/java/bin/../lib/jli/haswell/x86_64/libz.so.1
       214:	  trying file=/opt/java/bin/../lib/jli/haswell/libz.so.1
       214:	  trying file=/opt/java/bin/../lib/jli/x86_64/libz.so.1
       214:	  trying file=/opt/java/bin/../lib/jli/libz.so.1
       214:	  trying file=/opt/java/bin/../lib/tls/haswell/x86_64/libz.so.1
       214:	  trying file=/opt/java/bin/../lib/tls/haswell/libz.so.1
       214:	  trying file=/opt/java/bin/../lib/tls/x86_64/libz.so.1
       214:	  trying file=/opt/java/bin/../lib/tls/libz.so.1
       214:	  trying file=/opt/java/bin/../lib/haswell/x86_64/libz.so.1
       214:	  trying file=/opt/java/bin/../lib/haswell/libz.so.1
       214:	  trying file=/opt/java/bin/../lib/x86_64/libz.so.1
       214:	  trying file=/opt/java/bin/../lib/libz.so.1
       214:	 search cache=/usr/glibc-compat/etc/ld.so.cache
       214:	  trying file=/lib/libz.so.1
       214:
       214:	find library=libpthread.so.0 [0]; searching
       214:	 search path=/opt/java/bin/../lib/jli:/opt/java/bin/../lib		(RPATH from file /opt/java/bin/java)
       214:	  trying file=/opt/java/bin/../lib/jli/libpthread.so.0
       214:	  trying file=/opt/java/bin/../lib/libpthread.so.0
       214:	 search cache=/usr/glibc-compat/etc/ld.so.cache
       214:	  trying file=/usr/glibc-compat/lib/libpthread.so.0
       214:
       214:	find library=libjli.so [0]; searching
       214:	 search path=/opt/java/bin/../lib/jli:/opt/java/bin/../lib		(RPATH from file /opt/java/bin/java)
       214:	  trying file=/opt/java/bin/../lib/jli/libjli.so
       214:
       214:	find library=libdl.so.2 [0]; searching
       214:	 search path=/opt/java/bin/../lib/jli:/opt/java/bin/../lib		(RPATH from file /opt/java/bin/java)
       214:	  trying file=/opt/java/bin/../lib/jli/libdl.so.2
       214:	  trying file=/opt/java/bin/../lib/libdl.so.2
       214:	 search cache=/usr/glibc-compat/etc/ld.so.cache
       214:	  trying file=/usr/glibc-compat/lib/libdl.so.2
       214:
       214:	find library=libc.so.6 [0]; searching
       214:	 search path=/opt/java/bin/../lib/jli:/opt/java/bin/../lib		(RPATH from file /opt/java/bin/java)
       214:	  trying file=/opt/java/bin/../lib/jli/libc.so.6
       214:	  trying file=/opt/java/bin/../lib/libc.so.6
       214:	 search cache=/usr/glibc-compat/etc/ld.so.cache
       214:	  trying file=/usr/glibc-compat/lib/libc.so.6
       214:
       214:	find library=libc.musl-x86_64.so.1 [0]; searching
       214:	 search path=/opt/java/bin/../lib/jli:/opt/java/bin/../lib		(RPATH from file /opt/java/bin/java)
       214:	  trying file=/opt/java/bin/../lib/jli/libc.musl-x86_64.so.1
       214:	  trying file=/opt/java/bin/../lib/libc.musl-x86_64.so.1
       214:	 search cache=/usr/glibc-compat/etc/ld.so.cache
       214:	  trying file=/lib/libc.musl-x86_64.so.1
       214:
       214:
       214:	calling init: /usr/glibc-compat/lib/libpthread.so.0
       214:
       214:
       214:	calling init: /lib/libc.musl-x86_64.so.1
       214:
       214:
       214:	calling init: /usr/glibc-compat/lib/libc.so.6
       214:
       214:
       214:	calling init: /usr/glibc-compat/lib/libdl.so.2
       214:
       214:
       214:	calling init: /lib/libz.so.1
       214:
       214:
       214:	calling init: /opt/java/bin/../lib/jli/libjli.so
       214:
       214:
       214:	initialize program: /opt/java/bin/java
       214:
       214:
       214:	transferring control: /opt/java/bin/java
       214:
       214:	find library=libm.so.6 [0]; searching
       214:	 search path=/opt/java/bin/../lib/jli:/opt/java/bin/../lib		(RPATH from file /opt/java/bin/java)
       214:	  trying file=/opt/java/bin/../lib/jli/libm.so.6
       214:	  trying file=/opt/java/bin/../lib/libm.so.6
       214:	 search cache=/usr/glibc-compat/etc/ld.so.cache
       214:	  trying file=/usr/glibc-compat/lib/libm.so.6
       214:
       214:
       214:	calling init: /usr/glibc-compat/lib/libm.so.6
       214:
       214:
       214:	calling init: /opt/java/lib/server/libjvm.so
       214:
       214:	find library=librt.so.1 [0]; searching
       214:	 search path=/opt/java/bin/../lib/jli:/opt/java/bin/../lib		(RPATH from file /opt/java/bin/java)
       214:	  trying file=/opt/java/bin/../lib/jli/librt.so.1
       214:	  trying file=/opt/java/bin/../lib/librt.so.1
       214:	 search cache=/usr/glibc-compat/etc/ld.so.cache
       214:	  trying file=/usr/glibc-compat/lib/librt.so.1
       214:
       214:
       214:	calling init: /usr/glibc-compat/lib/librt.so.1
       214:
       214:
       214:	calling init: /opt/java/lib/libverify.so
       214:
       214:
       214:	calling init: /opt/java/lib/libjava.so
       214:
       214:	/opt/java/lib/server/libjvm.so: error: symbol lookup error: undefined symbol: JVM_begin_signal_setting (fatal)
       214:	find library=libnss_files.so.2 [0]; searching
       214:	 search path=/opt/java/bin/../lib/jli:/opt/java/bin/../lib		(RPATH from file /opt/java/bin/java)
       214:	  trying file=/opt/java/bin/../lib/jli/libnss_files.so.2
       214:	  trying file=/opt/java/bin/../lib/libnss_files.so.2
       214:	 search cache=/usr/glibc-compat/etc/ld.so.cache
       214:	  trying file=/usr/glibc-compat/lib/libnss_files.so.2
       214:
       214:
       214:	calling init: /usr/glibc-compat/lib/libnss_files.so.2
       214:
       214:
       214:	calling init: /opt/java/lib/libzip.so
       214:
       214:
       214:	calling init: /opt/java/lib/libjimage.so
       214:
       214:	/opt/java/lib/libjava.so: error: symbol lookup error: undefined symbol: Java_java_security_AccessController_doPrivileged (fatal)
       214:	/opt/java/lib/libjava.so: error: symbol lookup error: undefined symbol: Java_jdk_internal_reflect_Reflection_getCallerClass (fatal)
openjdk 11.0.7 2020-04-14 LTS
OpenJDK Runtime Environment Zulu11.39+15-CA (build 11.0.7+10-LTS)
OpenJDK 64-Bit Server VM Zulu11.39+15-CA (build 11.0.7+10-LTS, mixed mode)
[thread 219 also had an error]
#
[thread 220 also had an error]# A fatal error has been detected by the Java Runtime Environment:

#
#  SIGSEGV (0xb) at pc=0x00007f90b3e3da41, pid=214, tid=229
#
[thread 217 also had an error]
# JRE version: OpenJDK Runtime Environment (Zulu11.39+15-CA) (11.0.7+10) (build 11.0.7+10-LTS)
# Java VM: OpenJDK 64-Bit Server VM (11.0.7+10-LTS, mixed mode, tiered, compressed oops, g1 gc, linux-amd64)
# Problematic frame:
# C  [libc.so.6+0x11da41][thread 221 also had an error]

#
# No core dump will be written. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /usr/glibc-compat/lib/hs_err_pid214.log
[thread 215 also had an error]
#
# If you would like to submit a bug report, please visit:
#   http://www.azulsystems.com/support/
#
Aborted

@nadavwr
Copy link

nadavwr commented Jun 18, 2020

We've been able to get the JVM not to crash by lifting a prebuilt glibc-based zlib from another distro (forgot where I saw this, perhaps on a different ticket in this repo). However, we don't feel good about rolling that to production. Still looking for a better solution.

@prantlf
Copy link

prantlf commented Jul 11, 2020

If you do not need a very specific build of JDK, you can install Alpine packages for JDK 8 or 11:

FROM alpine:3.12
RUN apk --no-cache add openjdk8-jre
FROM alpine:3.12
RUN apk --no-cache add openjdk11-jre

@frol
Copy link
Member

frol commented Jul 13, 2020

Or just use the official https://hub.docker.com/_/openjdk image, which has Alpine tags.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
help wanted
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@nadavwr @frol @prantlf @xfh @menghuxiashan