[FIXED] Buildling boxplots incrementally from large datasets

Issue

Let s say i have 4 files saved on my computer as .npz files : W,X,Y and Z. Let s assume that my computer can not endure to load at the same time more than one of them in term of RAM consumption.

How can I be able to run this command ? :

 matplotlib.pyplot.boxplot([W],[X],[Y],[Z])

In other terms, how can I load W, plot W, delete W then load Y, plot Y, delete Y, ... and have the 4 of them on the same figure ? ( and not a subplot )

Thank you !

Solution

The matplotlib.axes.boxplot function actually calls two functions under the hood. One to compute the necessary statistics (cbook.boxplot_stats) and one to actually draw the plot (matplotlib.axes.bxp). You can exploit this structure, by calling the first for each dataset (by loading one at a time) and then feed the results to the plotting function.

In this example below we have 3 datasets and iterate over them to collect the output of cbook.boxplot_stats (which needs only very little memory). After that call to ax.bxp creates the graph. (In your application you would iteratively load a file, use boxplot_stats and delete the data)

import matplotlib.cbook as cbook
import matplotlib.pyplot as plt
import numpy as np


x = np.random.rand(10,10)
y = np.random.rand(10,10)
z = np.random.rand(10,10)

fig, ax = plt.subplots(1,1)

bxpstats = list()
for dataset, label in zip([x, y, z], ['X', 'Y', 'Z']):
    bxpstats.extend(cbook.boxplot_stats(np.ravel(dataset), labels=[label]))
ax.bxp(bxpstats)
plt.show()

Result:

Answered By - hitzg