[FIXED] When run a tensorflow session in iPython, GPU memory usage remain high when exiting iPython

Issue

I think it's some sort of bug.

The problem is quite simple:

1. launch ipython

2. import Tensorflow and run whatever session

3. type nvidia-smi in bash (see really high gpu memory usage, related process name, etc)

4. control+z quit ipython

5. type nvidia-smi in bash (still! really high GPU memory usage, and the same process name, strangely, these processes are not killed!)

I guess iPython failed to clean Tensorflow variables or graphs when exiting.

Is there any way I can clean the GPU memory without restart my machine?

System: Ubuntu 14.04
Python: Python3.5
IPython: IPython6.0.0

Solution

Control+Z doesn't quit a process, it stops it (use fg to bring it back up). If some computation is running in a forked process, it may not stop with the main process (I'm no OS guy, this is just my intuition).

In any case, properly quitting iPython (e.g. by Control+D or by running exit()) should solve the problem. If you need to interrupt a running command, first hit Control+C, then run exit().

Answered By - ales_t