[FIXED] Python Garbage Collection Cleanup Order On Exit

Issue

I have a context which stores information about the current state. This context is defined within Python. The context cleanup is handled using weakref's finalize() method.

class Context:
    def __init__(self, flags = 0):
        weakref.finalize(self, release_ctx, flags)
        self.flags = flags

This has worked totally fine up until this point. Recently I added a C extension which contains a C object which uses information from this context.

typedef struct {
    PyObject_HEAD
    C_CLASS* c_class_that_uses_info_from_py_ctx;
    PyObject* py_ctx;
} PyThing;

Within PyThing's c extension init method I increment the ref count of the py_ctx that is passed in and decrement the ref count on deallocation. I was under the impression that as long as an object has a non-zero ref count it wouldn't be garbage collected. This appears to not be the full story when the program exits. Basically the issue is I need to do some cleanup within the C object which relies upon the context. However, on program exit the context is being destroyed before the C object even though the context has a non-zero ref count (the weakref finalize function release_ctx() is being called before PyThing's dealloc method even though printing out the ref count of py_ctx via Py_REFCNT within the dealloc method shows that it still has 5 references). I found a statement here https://devguide.python.org/garbage_collector/ saying that if an object is unreachable all weak references are handled first. I think this is the cause for the Context being deallocated first but am not totally sure. Is there a way to prevent this deallocation from happing until all other objects have been deallocated? I've found a few articles describing Python's cyclic garbage collection but am unsure if it relates to this problem and how it might be used to solve it if it is.

Solution

After much trial and error I think I have a solution to the problem. I tried registering a cleanup function with the tp_finalize slot of PyThing but this didn't quite work. It seems that functions registered to weakref's finalize method take priority in terms of deallocation order on exit. Because Context is using weakref's finalize it gets cleaned up before PyThing on exit. I needed to register PyThing with weakref in order to get the correct deallocation order. PyThing 's deallocation method works as normal (i.e. it performs both a ref count decrement on the owned context as well as deallocates C_CLASS) but I also created a separate method to register PyThing with weakref (done within PyThing's init method in the C api) that only deallocates the C_CLASS. If the ref count gets to 0 in the middle of the program the deallocation method gets called as it should, however on exit the finalize method gets called and destroys the C_CLASS before the context is destroyed (the deallocation method may still be called so C_CLASS should be checked if it is NULL so delete/free isn't called a second time).

Answered By - Chris Uchytil