[FIXED] How to speed up of selecting data from a big array

Issue

--Hi, all

I have a big numpy array (named "Original_data") with the size of (512,512,600,51). I want to do boot strapping with the following codes

N_total_sig = 600
rand_Select_sig = np.random.randint(N_total_sig, size = N_total_sig)
sig = Origianl_data[:,:,rand_Select_sig,:]; 

But it is very slow with the above codes. Would you please tell me if there is a method to speed up of selecting data from a big array?

Thanks Erick

Solution

There is no direct solution to my problem, but there is an alternate way.

According to the suggestions by Jérôme Richard and Jérôme Richard, I try to avoid Copy 60 GiB of data in RAM. I did the following things.

1. Bootstraping to get the index of files.
2. load these files and calculate the mean on the fly.

By using the above strategy, the speed is still not fast. But the good thing is that I can run my codes in multiple workstations (These workstations don't have big RAM.)

Answered By - Erickzhou