Fatal error when stopping python session

[isubag]

Hi,

using latest (0.9) jpy version, I experienced an issue when stopping a python session from java.
With such trivial main file:

import org.jpy.*;

public class MainJPY {
    public static void main(String args[]){
        PyLib.startPython("/home/michele/anaconda2/bin");
        PyLib.stopPython(); 
    }
}

I get a fatal error when I call stopPython():

A fatal error has been detected by the Java Runtime Environment:

 SIGSEGV (0xb) at pc=0x00007f3b6f549092, pid=15482, tid=139893955520256

JRE version: Java(TM) SE Runtime Environment (8.0_91-b14) (build 1.8.0_91-b14)
Java VM: Java HotSpot(TM) 64-Bit Server VM (25.91-b14 mixed mode linux-amd64 compressed oops) Problematic frame:
C  [libpython2.7.so.1.0+0x11d092]  Py_Finalize+0x32

Is it a know issue?

[voutilad]

I've observed the same thing and looked into it a bit. Not sure if it's an issue or not...quite possibly is...but Py_Finalize is pretty finicky with Python native extensions according to the Python docs: Py_Finalize.

Looking at some other StackOverflow questions about seg faults from calling it, I'm not sure there's an easy solution other than to not call it. Plus, it could have different behavior on *nix vs. Windows based on using the Python shared library as well.

Sorry if this doesn't help answer the question, but I think it might help people understand the reason it seg faults (and possibly why it appears commented out in the current tests).

[forman]

Thanks for this. I hope to find some time soon for debugging into it. Does it happen on Windows too?

[voutilad]

It seems to be platform independent. The good news is I think I've found a very simple resolution for the segfault: call PyGILState_Ensure() before Py_Finalize().

I have PyLib.stopPython() now working in my fork of your master branch, but it's either unsurfaced sneaky bugs in the C bridge code or I've introduced one or many new bugs.

Test classes will run and pass individually, but grouping certain tests together in the same run causes some specific tests to fail. An example from running:

python3 setup.py clean && python3 setup.py bdist && mvn test -DargLine=-Xmx512m -Djpy.config=build/lib.macosx-10.11-x86_64-3.5/jpyconfig.properties -Djpy.debug=true -Dtest=UseCases,PyLibTest,PyObjectTest

...

testCreateProxyAndCallMultiThreaded(org.jpy.PyObjectTest)  Time elapsed: 0.004 sec  <<< ERROR!
java.lang.RuntimeException: Error in Python interpreter:
Type: <class 'ValueError'>
Value: cannot convert a Python 'str' to a Java 'java.lang.String'
Line: <not available>
Namespace: <not available>
File: <not available>
    at org.jpy.PyLib.callAndReturnValue(Native Method)
    at org.jpy.PyProxyHandler.invoke(PyProxyHandler.java:56)
    at com.sun.proxy.$Proxy6.initialize(Unknown Source)
    at org.jpy.PyObjectTest.testCallProxyMultiThreaded(PyObjectTest.java:244)
...

Any ideas? I'll continue to poke around and if I get things working I'll submit a PR.

[forman]

Thanks so much. Looking forward for your PR. What exactly have you changed? We'll try to reproduce the (new) problem here!

[voutilad]

I looked at it some more yesterday and my guess is that by adding PyLib.stopPython(); to the tests, and thus causing a "new"(?) interpreter to be started for each test, I'm seeing GIL deadlocks and possibly function/method premature garbage collection issues.

I've tried changing the test setup logic around, too, with no effect. Same errors when doing @BeforeClass/@AfterClass and @Before/@After. Yet, running the failing tests on their own...they pass! So very strange and that's why I'm convinced there's something going on in the native code controlling the Python interpreter.

[forman]

Now, I remember having the same issue at that time. I should have documented it. There is an issue with restarting the Python interpreter and then stopping it. The suggestion is then, not to restart the Python interpreter in your Java programs that use jpy. I'll mention this in the jpy Java API docs.

[voutilad]

Just submitted a PR containing what I believe to be a fix. This should let us start and stop the interpreter from Java without segfaults killing the JVM.