[FIXED] How to split numpy array in batches?

Issue

It sounds like easy not i dont know how to do.

i have numpy 2d array of

X = (1783,30)

and i want to split them in batches of 64. I write the code like this.

batches = abs(len(X) / BATCH_SIZE ) + 1  // It gives 28

I am trying to do prediction of results batchwise. So i fill the batch with zeros and i overwrite them with predicted results.

predicted = []

for b in xrange(batches): 

 data4D = np.zeros([BATCH_SIZE,1,96,96]) #create 4D array, first value is batch_size, last number of inputs
 data4DL = np.zeros([BATCH_SIZE,1,1,1]) # need to create 4D array as output, first value is  batch_size, last number of outputs
 data4D[0:BATCH_SIZE,:] = X[b*BATCH_SIZE:b*BATCH_SIZE+BATCH_SIZE,:] # fill value of input xtrain

 #predict
 #print [(k, v[0].data.shape) for k, v in net.params.items()]
 net.set_input_arrays(data4D.astype(np.float32),data4DL.astype(np.float32))
 pred = net.forward()
 print 'batch ', b
 predicted.append(pred['ip1'])

print 'Total in Batches ', data4D.shape, batches
print 'Final Output: ', predicted

But in the last batch number 28, there are only 55 elements instead of 64 (total elements 1783), and it gives

ValueError: could not broadcast input array from shape (55,1,96,96) into shape (64,1,96,96)

What is the fix for this?

PS: the network predictione requires exact batch size is 64 to predict.

Solution

I found a SIMPLE way of solving the batches problem by generating dummy and then filling up with the necessary data.

data = np.zeros(batches*BATCH_SIZE,1,96,96)
// gives dummy  28*64,1,96,96

This code will load the data exactly 64 batch size. The last batch will have dummy zeros at the end, but thats ok :)

pred = []
for b in batches:
 data4D[0:BATCH_SIZE,:] = data[b*BATCH_SIZE:b*BATCH_SIZE+BATCH_SIZE,:]
 pred = net.predict(data4D)
 pred.append(pred)

output =  pred[:1783] // first 1783 slice

Finally i slice out the 1783 elements from 28*64 total. This worked for me but i am sure there are many ways.

Answered By - pbu