[FIXED] Matplotlib sequentially printing /append to image

Issue

Currently, I have a huge dataset, which did not fit in memory. For training, I used the tf.Dataset to train my neural network. Unfortunately, now I want to verify my results but the dataset + printing do not fit into my 32 GB of memory. Is there a way to partially or sequential print a graph or multiple graphs to a file using matplotlib. So after each predict_on_batch I would draw a part of the graph (load image, draw, save image). In the end, I want a complete graph to verify my results. Any ideas or solutions to the "data do not fit in memory, but I need a graph to visualize it"-problem would be appreciated. It is not labeled data so there is no way besides checking it visually.

Solution

A way I would approach this problem is to save the information in a list instead of a partial plot.

I am assuming you want your pipeline to look like this:

1. create empty plot
2. predict on batch
3. plot new results
4. repeat steps 2 and 3

What I would suggest looks something like this:

1. initialize empty lists
2. predict on batch
3. append the results to lists
4. repeat steps 2 and 3
5. plot the final arrays

Answered By - Matei S