[FIXED] cannot release numpy array that previously output by jupyter notebook

Issue

I sometimes create large numpy array when using jupyer notebook, and found the memories may not be released as expected, if following the example:

# at cell [1]
import numpy as np
x = np.empty((int(1e10), ), dtype=float)
x[:] = 0
# at cell [2]
x   # so the cell output capture x
# at cell [3]
del x     # not released

However, the memories can be released if cell [2] is not executed. So I guess x is captured by the notebook kernel and its reference count is then kept to be non-zero.
Is there any way to explictly release such out-of-scope variables?

Solution

You can clear all output cache variables through the magic command reset and out parameter:

In [1]: x = np.empty(10 ** 10)

In [2]: x
Out[2]: array([0., 0., 0., ..., 0., 0., 0.])

In [3]: del x

In [4]: oldlocalskeys = set(locals())

In [5]: %reset -f out
Flushing output cache (1 entries)

In [6]: oldlocalskeys - locals().keys()
Out[6]: {'_2'}

Or if you only want to clear the object pointed to by a variable, you can use the magic command xdel, which will clear the cache variable reference together:

In [1]: x = np.empty(10 ** 10)

In [2]: x
Out[2]: array([0., 0., 0., ..., 0., 0., 0.])

In [3]: oldlocalskeys = set(locals())

In [4]: %xdel x

In [5]: oldlocalskeys - locals().keys()
Out[5]: {'_', '_2', 'x'}

To permanently disable output caching, you can set InteractiveShell.cache_size to 0 in the configuration file:

In [1]: []
[]

In [2]: {}
{}

In [3]: ()
()

In [4]: locals()
{...
 '_ih': ['', '[]', '{}', '()', 'locals()'],
 '_oh': {},
 '_dh': [WindowsPath('C:/windows/system32')],
 'In': ['', '[]', '{}', '()', 'locals()'],
 'Out': {},
 'get_ipython': <bound method InteractiveShell.get_ipython of <IPython.terminal.interactiveshell.TerminalInteractiveShell object at 0x000001F5B66304F0>>,
 'exit': <IPython.core.autocall.ExitAutocall at 0x1f5b6630e80>,
 'quit': <IPython.core.autocall.ExitAutocall at 0x1f5b6630e80>,
 '_': '',
 '__': '',
 '___': '',
 ...
 '_i': '()',
 '_ii': '{}',
 '_iii': '[]',
 '_i1': '[]',
 '_i2': '{}',
 '_i3': '()',
 '_i4': 'locals()'}

But note that this will also invalidate the underline cache. You cannot access the previous output results through 1/2/3 underline(s).

Answered By - Mechanic Pig