Possible bug building the new (4.2.2) version

[aghasemi]

Describe the bug
Hi. I noticed, after updating Python JEP version to 4.2.2, that sometimes(?) the pip build breaks with the following message, in an Ubuntu 24.04 container with default Python (3.10) and Java 21:

error: subprocess-exited-with-error
  × python setup.py bdist_wheel did not run successfully.
  │ exit code: 1
  ╰─> [29 lines of output]
      numpy not found, running without numpy support
      running bdist_wheel
      running build
      running setup_java
      Using JAVA_HOME: /usr/lib/jvm/java-21-openjdk-amd64
      running build_java
      /usr/lib/jvm/java-21-openjdk-amd64/bin/javac -deprecation -d build/java -h build/include -classpath src src/main/java/jep/Run.java src/main/java/jep/AllowPythonClassEnquirer.java src/main/java/jep/PyConfig.java src/main/java/jep/ClassEnquirer.java src/main/java/jep/PyMethod.java src/main/java/jep/JepConfig.java src/main/java/jep/DirectNDArray.java src/main/java/jep/Interpreter.java src/main/java/jep/NDArray.java src/main/java/jep/ClassList.java src/main/java/jep/Jep.java src/main/java/jep/JepAccess.java src/main/java/jep/LibraryLocator.java src/main/java/jep/NamingConventionClassEnquirer.java src/main/java/jep/JepException.java src/main/java/jep/AbstractNDArray.java src/main/java/jep/SubInterpreterOptions.java src/main/java/jep/SharedInterpreter.java src/main/java/jep/SubInterpreter.java src/main/java/jep/Proxy.java src/main/java/jep/MainInterpreter.java src/main/java/jep/python/PyBuiltins.java src/main/java/jep/python/InvocationHandler.java src/main/java/jep/python/PyCallable.java src/main/java/jep/python/MemoryManager.java src/main/java/jep/python/PyPointer.java src/main/java/jep/python/PyObject.java
      /usr/lib/jvm/java-21-openjdk-amd64/bin/javac -deprecation -d build/java/test -classpath build/java:src src/test/java/jep/test/TestCompiledScript.java src/test/java/jep/test/TestIteratorable.java src/test/java/jep/test/TestSharedModulesThreads.java src/test/java/jep/test/TestPyJObject.java src/test/java/jep/test/Test.java src/test/java/jep/test/TestGetCollectionBoxing.java src/test/java/jep/test/TestAllowPythonEnquirer.java src/test/java/jep/test/TestInvoke.java src/test/java/jep/test/TestGetJPyObject.java src/test/java/jep/test/TestSubInterpOptions.java src/test/java/jep/test/TestCloseWithThreads.java src/test/java/jep/test/TestGetTempValue.java src/test/java/jep/test/TestGetWithClass.java src/test/java/jep/test/TestRedirectStreams.java src/test/java/jep/test/TestOverload.java src/test/java/jep/test/TestSharedInterpreter.java src/test/java/jep/test/TestPyBuiltins.java src/test/java/jep/test/TestPyJType.java src/test/java/jep/test/TestSharedArgv.java src/test/java/jep/test/TestDefaultMethods.java src/test/java/jep/test/TestExec.java src/test/java/jep/test/TestPreInitVariables.java src/test/java/jep/test/TestExceptionCause.java src/test/java/jep/test/TestSharedModules.java src/test/java/jep/test/TestGetByteArray.java src/test/java/jep/test/synchronization/TestCrossLangSync.java src/test/java/jep/test/closeable/TestAutoCloseable.java src/test/java/jep/test/util/SubInterpreterThread.java src/test/java/jep/test/types/TestStaticFieldTypes.java src/test/java/jep/test/types/TestFieldTypes.java src/test/java/jep/test/types/TestMethodTypes.java src/test/java/jep/test/types/TestStaticMethodTypes.java src/test/java/jep/test/numpy/TestNumpyProdShared.java src/test/java/jep/test/numpy/TestNumpyArrayToString.java src/test/java/jep/test/numpy/TestNumpy.java
      [thread 55 also had an error]
      #
      # A fatal error has been detected by the Java Runtime Environment:
      #
      #  SIGSEGV (0xb) at pc=0x000079257cb4ce62, pid=40, tid=41
      #
      # JRE version: OpenJDK Runtime Environment (21.0.6+7) (build 21.0.6+7-Ubuntu-124.04.1)
      # Java VM: OpenJDK 64-Bit Server VM (21.0.6+7-Ubuntu-124.04.1, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, linux-amd64)
      # Problematic frame:
      # J 1741 c1 com.sun.tools.javac.tree.TreeTranslator.translate(Lcom/sun/tools/javac/tree/JCTree;)Lcom/sun/tools/javac/tree/JCTree; [email protected] (23 bytes) @ 0x000079257cb4ce62 [0x000079257cb4ce40+0x0000000000000022]
      #
      # Core dump will be written. Default location: /tmp/pip-install-ugwn04yy/jep_15fada7f64d347ec99efe726b29909b3/core
      #
      # An error report file with more information is saved as:
      # /tmp/pip-install-ugwn04yy/jep_15fada7f64d347ec99efe726b29909b3/hs_err_pid40.log
      [6.330s][warning][os] Loading hsdis library failed
      #
      # If you would like to submit a bug report, please visit:
      #   https://bugs.launchpad.net/ubuntu/+source/openjdk-21
      #
      error: command '/usr/lib/jvm/java-21-openjdk-amd64/bin/javac' failed with exit code -6
      [end of output]
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for jep

I couldn't reliably reproduce it. But I will continue investigating, and thought someone else may be interested. There is no problem when downgrading to 4.2.1

Thanks

To Reproduce
Run pip install jep==4.2.2

Expected behavior
JEP is installed

Environment (please complete the following information):

• OS Platform, Distribution, and Version: Ubuntu 24.04 LTS from Dockerhub
• Python Distribution and Version: 3.10, as is default on Ubuntu 24.04
• Java Distribution and Version: Adoptium(?) JDK 21
• Jep Version: 4.2.2
• Python packages used (e.g. numpy, pandas, tensorflow): pandas==2.2.3

Additional context

[bsteffensmeier]

It looks like the crash is in javac. There were some small changes to java files in Jep 4.2.2 but nothing major. It looks like the specific compile command that failed in your description was compiling test code and the only change to java test code in Jep 4.2.2 is the addition of the SubInterpreterThread class. Nothing in that class stands out to me as something that should be crashing the Java compiler.

I tried to reproduce your problem. I was unable to install python 3.10 on a ubuntu 24.04 container, as far as I can tell ubuntu 24.04 comes with python 3.12. Since the crash is in javac the python version shouldn't make much difference but perhaps I don't understand what environment you are in. Within the ubuntu:24.04 container I ran the following commands to test it and I did not see any crashing.

apt-get update
apt-get install openjdk-21-jdk python3-venv pip
python3 -m venv path/to/venv
source path/to/venv/bin/activate
for run in {1..100}; do echo $run; pip install --no-cache jep==4.2.2; pip uninstall -y jep==4.2.2; done

Please report back if you make any progress with your investigation or if you have any ideas how I might be able to recreate it.

[aghasemi]

Thanks for your investigation. Here is my Dockerfile that causes the failure. Sorry if I have mistaken Python version. I meant the one that is default on Ubuntu 24.04

FROM ubuntu:24.04
USER 0

ENV TZ Europe/Zurich
RUN echo $JAVA_HOME
RUN apt-get update && apt-get install -y --no-install-recommends build-essential openjdk-21-jdk-headless maven python3-pip python3-dev vim curl && rm -rf /var/lib/apt/lists/*
ENV JAVA_HOME /usr/lib/jvm/java-21-openjdk-amd64


#USER 1000
RUN pip3 install --no-cache-dir --break-system-packages pandas==2.2.3 jep==4.2.2

I use corporate mirrors of repositories (Dockerhub + PIP_INDEX_URL), but I doubt that has an effect, maybe? :-/

And it is, as I said, interestingly hit and miss still, with more success than failure, it seems.

[bsteffensmeier]

Thanks for the Dockerfile. I've built it 30 times on my machine and it succeeded each time. After building a few times I changed the Dockerfile to add an ARG right before the pip install so I could increment the arg and prevent docker from caching the last step. I've included my Dockerfile below for reference.

FROM ubuntu:24.04
USER 0

ENV TZ Europe/Zurich
RUN echo $JAVA_HOME
RUN apt-get update && apt-get install -y --no-install-recommends build-essential openjdk-21-jdk-headless maven python3-pip python3-dev vim curl && rm -rf /var/lib/apt/lists/*
ENV JAVA_HOME /usr/lib/jvm/java-21-openjdk-amd64


#USER 1000
ARG repeat=1
RUN pip3 install --no-cache-dir --break-system-packages pandas==2.2.3 jep==4.2.2

I do not have any good explanation why it intermittently fails for you and not me. The best idea I can come up with is that perhaps your corporate mirror has an older version of ubuntu 24.04 with a bug. But since javac is crashing I would think the bug would be in javac and that is installed by apt-get, not included on the docker image. If your corporate mirror has an apt repository maybe you have a buggy version of javac but this all seems very unlikely.

I don't think there is anything else I can do to troubleshoot this but I am curious to see if anyone else sees the problem.

Please let us know if you find any new details about this problem.

[aghasemi]

Thanks a lot. Yes it could be the corporate mirror in that sense. I hadn't thought about multiple 24.4 variants of Ubuntu/Java.

[aghasemi]

So, I have in the container Java 21.0.6 and Ubuntu 24.04.01, both apparently behind the latest minor/patch releases. Could be the cause. Thanks a lot for the hint.

Best