[FIXED] How to initialise a fixed-size ListArray in pyarrow from a numpy array efficiently?

Issue

How would I efficiently initialise a fixed-size pyarray.ListArray from a suitably prepared numpy array?

The documentation of pyarray.array indicates that a nested iterable input structure works, but in practice that does not work if the outer iterable is a numpy array:

import numpy as np
import pyarrow as pa

n = 1000
w = 3

data = np.arange(n*w,dtype="i2").reshape(-1,w)

# this works:
pa.array(list(data),pa.list_(pa.int16(),w))
# this fails:
pa.array(data,pa.list_(pa.int16(),w))
# -> ArrowInvalid: only handle 1-dimensional arrays

It seems ridiculus to split an input array directly matching the Arrow specification into n separate arrays and then re-assemble from there.

pyarray.ListArray.from_arrays seems to require an offsets argument, which only has a meaning for variable-size lists.

Solution

I believe you are looking for pyarrow.FixedSizeListArray.from_arrays which, regrettably, appears undocumented (I went ahead and filed a JIRA ticket)

You'll want to reshape your numpy array as a contiguous array first.

import numpy as np
import pyarrow as pa

len = 10
width = 3
# Or just skip the initial reshape but keeping it in to simulate real data
arr = np.arange(len*width,dtype="i2").reshape(-1,width)
arr.shape = -1
pa.FixedSizeListArray.from_arrays(arr, width)

Answered By - Pace