Multithreading frequent calls is crashing

[Envenger]

If i call the JSON input function around once every second, its crashing the game with a large python log.
This happens on the CPU version of the plugin but I don't call any tensorflow commands so i doubt that would be causing it.

My JSON input function doesn't do anything for now, just returns an empty quote.

    def onJsonInput(self, jsonInput):
        result = ""
        return result

This crash doesn't occur with multi threading off.
Attached the crash logs.
ShooterAIProject-backup-2019.04.15-14.17.37.log

[Envenger]

Do you want any help in replicating the bug? I can share you a small project where you can replicate the bug.

[getnamo]

That would be helpful! I suspect some race condition is occurring, but that would be weird because python uses a global interpreter lock, perhaps I'm not doing something quite right.

Note to self: c++ function used for this is found at https://github.com/getnamo/UnrealEnginePython/blob/master/Source/UnrealEnginePython/Private/UEPyEngine.cpp#L814

Note2: this is new https://github.com/20tab/UnrealEnginePython/blob/0393f40181988789eeec95d1cd9d6eec811ec2a2/android/python27/include/ceval.h#L79, do we need to wrap our function with it?

[magomedb]

I have a similar issue where using multithreading with similarily frequent calls to the JSON input function result in the following error occuring repeatedly inside the output log.

File "C:\Users\User\Documents\Unreal Projects\IAF\IAF\Plugins\tensorflow-ue4\Content\Scripts\TensorFlowComponent.py", line 116, in json_input_blocking
if(self.uobject.ShouldUseMultithreading):
Exception: PyUObject is in invalid state

The engine crashes after a while and the crash log lists python36, kernel32 and ntdll. Can also see that the memory gets gradually filled over time while it's running, until the eventual crash. Turning multithreading off removes the errors and doesn't result in a crash, and the memory doesn't get filled either.

Assuming that this is a similar problem to the original, wouldn't this indicate that the problem is a memory leak?

[getnamo]

With multi-threading on each call to Json input gets handled by a different thread. What's likely happening is a second thread tries to touch the same data before the first one is done with it and it unravels the memory by accessing out of bounds data (despite GIL or in the TF layer), then the continued calls keep leaking (it should have crashed earlier). Probably the best way forward would be to figure out how to add a lock so that JSON inputs get queued one at a time, not starting a new input until last one is fully done. Optionally a single JSON input thread would be the more efficient solution (with internal event queue), but I'm unsure if I know how to properly do that atm.

Thanks for providing more examples of the bug, it does narrow down the potential source.

[Uperstream]

I have encountered the same problem. I requested json input event every tick as well.
LogPython: Error: Exception in thread Thread-2291:
Traceback (most recent call last):
File "threading.py", line 916, in _bootstrap_inner
File "threading.py", line 864, in run
File "F:\ProjectFile\UnrealProject\AItest\Plugins\UnrealEnginePython\Content\Scripts\upythread.py", line 19, in backgroundAction
result = action(actionArgs)
File "F:\ProjectFile\UnrealProject\AItest\Plugins\tensorflow-ue4\Content\Scripts\TensorFlowComponent.py", line 116, in json_input_blocking
if(self.uobject.ShouldUseMultithreading):
Exception: PyUObject is in invalid state

I got a different error when I turned off the multithreading:

LogPython: Error: Variable pi/dense/kernel already exists, disallowed. Did you mean to set reuse=True or reuse=tf.AUTO_REUSE in VarScope? Originally defined at:
File "F:\ProjectFile\UnrealProject\AItest\Plugins\UnrealEnginePython\Binaries\Win64\Lib\site-packages\tensorflow\python\framework\ops.py", line 1770, in init
self._traceback = tf_stack.extract_stack()
File "F:\ProjectFile\UnrealProject\AItest\Plugins\UnrealEnginePython\Binaries\Win64\Lib\site-packages\tensorflow\python\framework\ops.py", line 3274, in create_op
op_def=op_def)
File "F:\ProjectFile\UnrealProject\AItest\Plugins\UnrealEnginePython\Binaries\Win64\Lib\site-packages\tensorflow\python\util\deprecation.py", line 488, in new_func
return func(*args, **kwargs)
LogPython: Error: Traceback (most recent call last):
LogPython: Error: File "F:\ProjectFile\UnrealProject\AItest\Plugins\tensorflow-ue4\Content\Scripts\TensorFlowComponent.py", line 143, in setup_complete
self.train()
LogPython: Error: File "F:\ProjectFile\UnrealProject\AItest\Plugins\tensorflow-ue4\Content\Scripts\TensorFlowComponent.py", line 73, in train
self.train_blocking()
LogPython: Error: File "F:\ProjectFile\UnrealProject\AItest\Plugins\tensorflow-ue4\Content\Scripts\TensorFlowComponent.py", line 152, in train_blocking
self.trained = self.tfapi.onBeginTraining()
LogPython: Error: File "F:\ProjectFile\UnrealProject\AItest\Content\Scripts\DPPOnothread.py", line 194, in onBeginTraining
self.model = PPO(epMax, 8, 5, 1e-4, 2e-4, self.que)
LogPython: Error: File "F:\ProjectFile\UnrealProject\AItest\Content\Scripts\DPPOnothread.py", line 36, in init
pi, pi_params = self._build_anet('pi', trainable=True)
LogPython: Error: File "F:\ProjectFile\UnrealProject\AItest\Content\Scripts\DPPOnothread.py", line 68, in _build_anet
l1 = tf.layers.dense(self.tfs, 200, tf.nn.relu, trainable=trainable)
LogPython: Error: File "F:\ProjectFile\UnrealProject\AItest\Plugins\UnrealEnginePython\Binaries\Win64\Lib\site-packages\tensorflow\python\layers\core.py", line 184, in dense
return layer.apply(inputs)
LogPython: Error: File "F:\ProjectFile\UnrealProject\AItest\Plugins\UnrealEnginePython\Binaries\Win64\Lib\site-packages\tensorflow\python\keras\engine\base_layer.py", line 817, in apply
return self.call(inputs, *args, **kwargs)
LogPython: Error: File "F:\ProjectFile\UnrealProject\AItest\Plugins\UnrealEnginePython\Binaries\Win64\Lib\site-packages\tensorflow\python\layers\base.py", line 374, in call
outputs = super(Layer, self).call(inputs, *args, **kwargs)
LogPython: Error: File "F:\ProjectFile\UnrealProject\AItest\Plugins\UnrealEnginePython\Binaries\Win64\Lib\site-packages\tensorflow\python\keras\engine\base_layer.py", line 746, in call
self.build(input_shapes)
LogPython: Error: File "F:\ProjectFile\UnrealProject\AItest\Plugins\UnrealEnginePython\Binaries\Win64\Lib\site-packages\tensorflow\python\keras\layers\core.py", line 944, in build
trainable=True)
LogPython: Error: File "F:\ProjectFile\UnrealProject\AItest\Plugins\UnrealEnginePython\Binaries\Win64\Lib\site-packages\tensorflow\python\layers\base.py", line 288, in add_weight
getter=vs.get_variable)
LogPython: Error: File "F:\ProjectFile\UnrealProject\AItest\Plugins\UnrealEnginePython\Binaries\Win64\Lib\site-packages\tensorflow\python\keras\engine\base_layer.py", line 609, in add_weight
aggregation=aggregation)
LogPython: Error: File "F:\ProjectFile\UnrealProject\AItest\Plugins\UnrealEnginePython\Binaries\Win64\Lib\site-packages\tensorflow\python\training\checkpointable\base.py", line 639, in _add_variable_with_custom_getter
**kwargs_for_getter)
LogPython: Error: File "F:\ProjectFile\UnrealProject\AItest\Plugins\UnrealEnginePython\Binaries\Win64\Lib\site-packages\tensorflow\python\ops\variable_scope.py", line 1487, in get_variable
aggregation=aggregation)
LogPython: Error: File "F:\ProjectFile\UnrealProject\AItest\Plugins\UnrealEnginePython\Binaries\Win64\Lib\site-packages\tensorflow\python\ops\variable_scope.py", line 1237, in get_variable
aggregation=aggregation)
LogPython: Error: File "F:\ProjectFile\UnrealProject\AItest\Plugins\UnrealEnginePython\Binaries\Win64\Lib\site-packages\tensorflow\python\ops\variable_scope.py", line 540, in get_variable
aggregation=aggregation)
LogPython: Error: File "F:\ProjectFile\UnrealProject\AItest\Plugins\UnrealEnginePython\Binaries\Win64\Lib\site-packages\tensorflow\python\ops\variable_scope.py", line 492, in _true_getter
aggregation=aggregation)
LogPython: Error: File "F:\ProjectFile\UnrealProject\AItest\Plugins\UnrealEnginePython\Binaries\Win64\Lib\site-packages\tensorflow\python\ops\variable_scope.py", line 861, in _get_single_variable
name, "".join(traceback.format_list(tb))))
LogPython: Error: ValueError: Variable pi/dense/kernel already exists, disallowed. Did you mean to set reuse=True or reuse=tf.AUTO_REUSE in VarScope? Originally defined at:
File "F:\ProjectFile\UnrealProject\AItest\Plugins\UnrealEnginePython\Binaries\Win64\Lib\site-packages\tensorflow\python\framework\ops.py", line 1770, in init
self._traceback = tf_stack.extract_stack()
File "F:\ProjectFile\UnrealProject\AItest\Plugins\UnrealEnginePython\Binaries\Win64\Lib\site-packages\tensorflow\python\framework\ops.py", line 3274, in create_op
op_def=op_def)
File "F:\ProjectFile\UnrealProject\AItest\Plugins\UnrealEnginePython\Binaries\Win64\Lib\site-packages\tensorflow\python\util\deprecation.py", line 488, in new_func
return func(*args, **kwargs)

I have seen this PyUObject is in invalid state error has been mentioned in the original repository. I don't really understand it. I'll post the link below.

https://github.com/20tab/UnrealEnginePython/blob/master/docs/MemoryManagement.md

[Uperstream]

I have fixed the error when I turned off multithreading. It works when I turned it off.