[FIXED] Is there a max size of samples that tensorflow Datasets can hold?

Issue

I have a few questions regarding tensorflow Datasets.

I've created a tensorflow dataset object in a jupyter notebook, a training data set holding 3.7 million samples and a test set holding close to 1 million samples. Each sample is a time series of 1000 values.

For both the train and test set I did

dataset = dataset.cache()
dataset = dataset.shuffle(len(dataset))
dataset = dataset.batch(batch_size)
dataset = dataset.prefetch(tf.data.experimental.AUTOTUNE)

The creation of these cells of the notebook are executed splendidly. But right in the next cell, when I want to enumerate through the test data set, my jupyter kernel connection breaks (also described here. The debug output revealed this problem:

tensorflow/core/framework/cpu_allocator_impl.cc:80] Allocation of 18936000000 exceeds 10% of free system memory.

This issue is also described here. I've already got the batch_size down to 1, I enabled

physical_devices = tf.config.list_physical_devices('GPU')
tf.config.experimental.set_memory_growth(physical_devices[0], True)

but I still have this problem. My machine has 64GBs of RAM and 6GB of GPU memory.

My questions are:

1. Is there a limit to how many samples tensorflow datasets can hold?
2. I could decrease the time series length down further, but would this even help if there is such a limit?
3. Is there any intelligent idea/way (or other more efficient data structure) to not run into this memory error?
4. Is this the GPU memory or the RAM memory that is exceeded?

Sorry for this whole bunch of questions, Im learning and Im grateful for anything.

Solution

The reason you're overloading the memory is that you're fitting all of it in memory when you cache() and set the shuffle() buffer to be the entire dataset. If you remove the former and massively decrease the shuffle buffer it should help.

tensorflow/core/framework/cpu_allocator_impl.cc:80] Allocation of 18936000000 exceeds 10% of free system memory.

It's the CPU

Answered By - Nicolas Gervais