[FIXED] In Python fast way to convert a bytearray to array of tuples (Without using NumPy)

Issue

I need to speed up the following code:-

index = 0
WIDTH = 1920
HEIGHT = 1080
for o in xrange(WIDTH * HEIGHT):
    outbuf[o] = (buffer[index], buffer[index+1], buffer[index+2])
    index += 3

Which converts a bytearray to an array of tuples, but is too slow, is there any way to make it faster without using numpy.

Solution

Building outbuf using the grouper recipe from itertools is almost twice as fast on my machine.

This is the grouper function from the docs:

def grouper(iterable, n, fillvalue=None):
    "Collect data into fixed-length chunks or blocks"
    # grouper('ABCDEFG', 3, 'x') --> ABC DEF Gxx
    args = [iter(iterable)] * n
    return izip_longest(fillvalue=fillvalue, *args)

This script compares timings for the function in the question, using grouper to populate outbuf and using grouper to return outbuf rather than populating an external list. The fastest method is to return outbuf from the function.

from itertools import izip_longest
import random
import sys
import timeit

WIDTH = 1920
HEIGHT = 1080


buffer_ = bytearray(random.randint(0, 255) for _ in xrange(WIDTH * HEIGHT * 3))

# Function from the question
def extract(buffer_):
    index = 0
    for o in xrange(WIDTH * HEIGHT):
        outbuf[o] = (buffer_[index], buffer_[index+1], buffer_[index+2])
        index += 3


def grouper(iterable, n, fillvalue=None):
    args = [iter(iterable)] * n
    return izip_longest(fillvalue=fillvalue, *args)


# Use grouper with external outbuf
def grouper_extract(buffer_):
    groups = grouper(buffer_, 3)
    for o in xrange(WIDTH * HEIGHT):
        outbuf[o] = next(groups)


# Return a list using grouper
def grouper_list(buffer_):
    return list(grouper(buffer_, 3))


if __name__ == '__main__':
    # Set number of timeit repetitions.
    try:
        number = int(sys.argv[1])
    except IndexError:
        number = 50
    outbuf = [0 for _ in xrange(WIDTH * HEIGHT * 3)]
    print 'OP function'
    print timeit.timeit(setup="from __main__ import extract, outbuf, buffer_",
                        stmt="extract(buffer_)", number=number)
    print
    print 'Fill external outbuf with grouper'
    print timeit.timeit(setup="from __main__ import grouper_extract, outbuf, buffer_",
                        stmt="grouper_extract(buffer_)", number=number)
    print
    print 'Return outbuf using grouper'
    print timeit.timeit(setup="from __main__ import grouper_list, buffer_",
                        stmt="outbuf = grouper_list(buffer_)", number=number)
    print

Here are the timings for 50 repetitions of each approach:

OP function
39.3345730305

Fill external outbuf with grouper
30.0249710083

Return outbuf using grouper
20.357350111

Answered By - snakecharmerb