[FIXED] Segmentation fault (core dumped) with optuna

Issue

I know this question has already been asked multiple times, but with the answers of the other threads, I couldn't solve my problem. I'm running a python script in the command line with python3 CTGAN_noscale.py --database_name CTGAN_noshift and receive the following error (with faulthandler):

Fatal Python error: Segmentation fault

Current thread 0x00007f57e97fe700 (most recent call first):
<no Python frame>

Thread 0x00007f593db07740 (most recent call first):
  File "/usr/local/lib/python3.8/dist-packages/torch/autograd/__init__.py", line 145 in backward
  File "/usr/local/lib/python3.8/dist-packages/torch/tensor.py", line 245 in backward
  File "/usr/local/lib/python3.8/dist-packages/ctgan/synthesizers/ctgan.py", line 374 in fit
  File "CTGAN_noscale.py", line 140 in objective
  File "CTGAN_noscale.py", line 162 in <lambda>
  File "/usr/local/lib/python3.8/dist-packages/optuna/_optimize.py", line 216 in _run_trial
  File "/usr/local/lib/python3.8/dist-packages/optuna/_optimize.py", line 162 in _optimize_sequential
  File "/usr/local/lib/python3.8/dist-packages/optuna/_optimize.py", line 65 in _optimize
  File "/usr/local/lib/python3.8/dist-packages/optuna/study.py", line 401 in optimize
  File "CTGAN_noscale.py", line 162 in run_CTGAN
  File "CTGAN_noscale.py", line 210 in <module>
Segmentation fault (core dumped)

It seems that the problem is somehow with optuna. The weird part is that everything worked fine on another server, after changing the server it crashed like this

Update

I found out that the the problem doesn't occur when I don't use a docker container OR use a docker container without GPU.

Solution

I solved the problem by rebuilding a new image and derived a container from this image. In this container somehow the error didn't appear anymore.

Answered By - zepp_e