[FIXED] Python: memory corruption after successful return from a ctypes foreign function

Issue

Non-essential background: I am having a multitude of problems with ctypes pointers that I have so far been able to work around by stripping everything down to a c_void_p on the Python side, then digging out what I need in C via recasting and pointer arithmetic. However, I've finally hit a wall in Python that I can't work around and I'm hoping that the sage answers to this MWE allow me to fix all of my other hacks.

The MWE: In both of the cases below, I successfully define, populate, pass, and return a struct from/to Python to/from a simple C foreign function. In the first case (ff1.py), everything is fine. In the second case (ff2.py), the function returns and successfully dereferences in Python, then Python crashes. The difference between the two cases is whether the struct that is passed out from Python is imported as a variable from a module or whether it is returned from a Python function before being passed to C.

ff.h:

struct dummy
{
  int      dim1;
  int      dim2;
  double * array_ptr;
};

void
foreign_func( struct dummy * ptr_to_dummy );

ff.c:

#include "ff.h"

__attribute__((visibility("default")))
void
foreign_func( struct dummy * d_ptr )
{
  const int N = d_ptr->dim1;
  const int M = d_ptr->dim2;

  for( int i=0; i<N; i++ )
    for( int j=0; j<M; j++ )
      d_ptr->array_ptr[ i*M + j ] *= 3.14159;

  return;
}

struct_wo_funcs.py:

import numpy as np
import ctypes

class dummy_struct_type( ctypes.Structure ):
    _fields_ = [ ( 'dim1',      ctypes.c_int ),
                 ( 'dim2',      ctypes.c_int ),
                 ( 'array_ptr', ctypes.POINTER( ctypes.c_double ) ) ]

n = 12
m = 50
a = np.ones( (n,m), dtype=ctypes.c_double, order='C' )

dummy_struct_instance = \
        dummy_struct_type( ctypes.c_int( n ),
                           ctypes.c_int( m ),
                           a.ctypes.data_as( ctypes.POINTER( ctypes.c_double ) ) )

struct_w_funcs.py:

import numpy as np
import ctypes

class dummy_struct_type( ctypes.Structure ):
    _fields_ = [ ( 'dim1',      ctypes.c_int ),
                 ( 'dim2',      ctypes.c_int ),
                 ( 'array_ptr', ctypes.POINTER( ctypes.c_double ) ) ]

def make_instance( n, m ):
    a = np.ones( (n,m), dtype=ctypes.c_double, order='C' )

    return dummy_struct_type( ctypes.c_int( n ),
                              ctypes.c_int( m ),
                              a.ctypes.data_as( ctypes.POINTER( ctypes.c_double ) ) )

ff1.py:

import numpy as np
import ctypes
import os
from struct_wo_funcs import dummy_struct_type, dummy_struct_instance

fflib = ctypes.CDLL( os.path.abspath( 'ff_lib.so' ) )

fflib.foreign_func.argtypes = [ ctypes.POINTER( dummy_struct_type ) ]
fflib.foreign_func.restype  = None

fflib.foreign_func( ctypes.byref( dummy_struct_instance ) )

print( dummy_struct_instance.array_ptr[19] )

ff2.py:

import numpy as np
import ctypes
import os
from struct_w_funcs import dummy_struct_type, make_instance

fflib = ctypes.CDLL( os.path.abspath( 'ff_lib.so' ) )

fflib.foreign_func.argtypes = [ ctypes.POINTER( dummy_struct_type ) ]
fflib.foreign_func.restype  = None

dummy_struct_instance = make_instance( 12, 50 )

fflib.foreign_func( ctypes.byref( dummy_struct_instance ) )

print( dummy_struct_instance.array_ptr[19] )

I have tried this with a variety of compilers on Linux. I have also changed the export and compiled with gcc/MinGW and cl/MSVC on Windows, all to the same effect. (To be fair, running at the MSVC developer prompt does not result in an outright crash; rather, Windows flinches for a while, then recovers back to the prompt without reporting anything.) Here is a representative Linux crash with gcc v7.5.0 and Python 3.6.9:

$ gcc -fPIC -c ff.c && gcc -fPIC -shared -o ff_lib.so ff.o

$ python3 ff1.py
3.14159

$ python3 ff2.py
3.14159
free(): corrupted unsorted chunks
Aborted (core dumped)

I have valgrinded the heck out of this and can only learn that something is wrong in the guts of Python. Ultimately, I want to be able to use functions in Python to generate the structs that I pass out; that is, I wish ff2.py would not crash. If your answer is simply another work-around (like some magic combination of numpy declarations and byrefs, etc.), that is good enough for me. As far as I know, this is a problem with Python and not C, but if you can get it to work by altering the C side of the FFI, I'd love to see how.

Solution

A ctypes pointer to your array data does not keep the array alive. You can make the first one crash with by just del a.

Create a wrapper for the struct that also keeps the numpy array alive by holding a reference to it.

Answered By - Antti Haapala -- Слава Україні