[FIXED] Is there a faster way to read, resort and convert a binary file?

Issue

I have a binary file that contains a certain number of samples and each sample contains four bytes. The data is acquired via 14 bit ADCs and I have the following bit assignment: b31-b29=Not Used, b28=Digital input, b27-b14=chB(signed), b13-b0=chA(signed). In the end I want to perform a FFT for chA and chB. To get there I use the following Python code:

1. Here the binary data file is opened as a bitstring and the samples, i.e. 516x1024x32x32 bits are read and appended to a bit array. This is done by reading one sample (4 bytes or 32 bits) at a time, inverting the byte order and then putting this bitstring in a bit array. This is repeated for all samples:

swap = BitArray()

f = ConstBitStream(filename='data.kbin')
f.pos = 0
samples = 516*1024*32
sample_rate = 30517.578125

for i in range(samples):
    data = BitArray()
    g = f.read('bits:32')
    data.append(g)
    data.byteswap(4)
    swap.append(data)

2. The newly ordered array is again opened as bitstring:

data2 = ConstBitStream(swap)

3. The bitstring is now read in a way so that the correct bit assignment (as shown above) is applied and each bitstring is converted to a signed integer. Also each integer referring to chA and chB are put into the corresponding list:

chA = []
chB = []

data2.pos = 0
for i in range(samples):
    a = data2.read('int:3')
    b = data2.read('int:1')
    c = data2.read('int:14')
    d = data2.read('int:14')
    chA.append(d)
    chB.append(c)

4. Calculating the FFT:

dt = 1 / sample_rate

yf_A = fftpack.rfft(chA)
yf_B = fftpack.rfft(chB)
xf = fftpack.rfftfreq(samples, dt)

This code works and I get the desired result but it takes really long. The first step takes almost 10 min and the third about 3 min. I am quite new to Python so my knowledge is rather small. How can I speed things up? Thanks.

Solution

I found a much faster way:

# input the path t the data file
file_kbin = input("Path to data file: ")

# initialize two lists where the converted data for chA and chB will be stored

CHA = []
CHB = []

# get the number of samples in the data file

size = 516 * 1024 * 32

# here the binary file is opened and converted following the byte assignment given above
with open(file_kbin, 'rb') as data:

    data.seek(0)                                          # make sure to start at the beginning of the file

    for i in range(size):                                 # loop over the complete file

        sample = data.read(4)                             # read one sample (byte0 byte1 byte2 byte3) 
        tmp = int.from_bytes(sample, byteorder='little')  # store the data as integer following the byteorder little (byte3 byte2 byte1 byte0)          

        chA = tmp & 0x00003FFF                            # set chA to byte1 byte0 with b15 and b14 = 0
        if (chA & 0x2000):                                # check if chA negative
            chA = chA - 0x4000                            # get the correct negative value

        chB = (tmp & 0x0FFFc000) >> 14                    # set all bits to 0 except for b27-b14 and shift right by 14 bits
        if (chB & 0x2000):                                # check if chB negative
            chB = chB - 0x4000                            # get the correct negative value

        CHA.append(chA)                                   # store the values in the corresponding list
        CHB.append(chB)

(Using the corresponding code in C++ would again be a lot faster.)

Answered By - Baal