[FIXED] How do I convert a pointer returned by a C function invoked using ctypes into a numpy array?

Issue

I have a function in C which returns an array of data and its length.

Everything compiles and works fine and I can invoke the function.

>>> from ctypes import *
>>> import numpy as np
>>> test = cdll.LoadLibrary("/home/user/test_ctypes_1/testlib.so")
>>> length = c_int()
>>> data = c_void_p()
>>> test.get_uint_array(byref(data), byref(length))

The notebook returns:

>>> print(length); print(data)
c_int(10)
c_void_p(34370499648)

I assume I have received the memory address for the allocated C array.

---

Now the problem I face is that I do not know how can I turn the c_void_p into a numpy numpy.uint32 array.

---

This is what I have tried:

a = np.ctypeslib.as_array(data, shape=(length,))

---

This is the error I get:

ValueError                                Traceback (most recent call last)
/usr/local/lib/python3.7/site-packages/numpy/core/_internal.py in _dtype_from_pep3118(spec)
    565     stream = _Stream(spec)
--> 566     dtype, align = __dtype_from_pep3118(stream, is_subdtype=False)
    567     return dtype

/usr/local/lib/python3.7/site-packages/numpy/core/_internal.py in __dtype_from_pep3118(stream, is_subdtype)
    642         else:
--> 643             raise ValueError("Unknown PEP 3118 data type specifier %r" % stream.s)
    644 

ValueError: Unknown PEP 3118 data type specifier 'P'

The above exception was the direct cause of the following exception:

ValueError                                Traceback (most recent call last)
<ipython-input-10-383866f696da> in <module>
----> 1 a = np.ctypeslib.as_array(data, shape=(length,))

/usr/local/lib/python3.7/site-packages/numpy/ctypeslib.py in as_array(obj, shape)
    521             obj = ctypes.cast(obj, p_arr_type).contents
    522 
--> 523         return array(obj, copy=False)
    524 
    525 

ValueError: '<P' is not a valid PEP 3118 buffer format string

---

This is the C code (included as reference):

This is my c_lib.c with the minimum reproducible example:

#include <stdlib.h>

void get_uint_array(unsigned int **data, int *length);

void get_uint_array(unsigned int **data, int *length) {
        int i;
        *length = 10;
        *data = malloc(*length*sizeof(unsigned int));
        for(i=0; i<*length; i++) {
                (*data)[i] = i+1;
        }
}

---

This is the makefile (included as reference):

CC = clang
CCFLAGS = -c -g -std=c99 -Wall -Werror -fPIC
SHAREDLINKFLAGS = -std=c99 -Wall -Werror -shared
INCLUDE = -I/usr/local/include

# SHARED LIBRARY (C)
testlib.so:     c_lib.o
                ${CC} ${SHAREDLINKFLAGS} -o testlib.so c_lib.o

# OBJECT MODULES
c_lib.o:        c_lib.c
                ${CC} ${CCFLAGS} c_lib.c ${INCLUDE}

---

I am using clang under FreeBSD 12 in case that might be relevant (I guess it is not).

Solution

I finally found the way:

# CALLING C FUNCTION, RETURNS unsigned int *c_data AND int c_length
c_length = c_int()
c_data = c_void_p()
test.get_uint_array(byref(c_data), byref(c_length));

# CONVERT TO NUMPY
data = np.ctypeslib.as_array(cast(c_data, POINTER(c_uint)), shape=(c_length.value,))

Notes:

• Data (c_data) must be released later in C using free and the c_data pointer.
• Note how the function returns unsigned int * and we handle a void * and later cast into a pointer. Not sure if there is a way to directly receive an unsigned int * pointer and avoid the cast.
• adding .copy() to the end of the .as_array method copies the array to the numpy object and data memory allocated by C can be immediately released (so now the memory is handled by Python). This might be useful in certain scenarios but duplicates the memory and there is the overhead of copying data into the new array.

I am still wondering if this is the optimal way of achieving this data conversion. If anyone knows another way to make this more efficient feel free to add comments or post another answer.

Answered By - M.E.