[FIXED] customizing typing.NamedTuple

Issue

I'm using NamedTuples to hold data, and I want to add a method that can be inherited by multiple NamedTuple based classes. But when I try using multiple inheritance or subclassing NamedTuple based classes, it doesn't work. Specifically, I'm trying to automatically give all of my data classes a method that can look at the classes annotations and then call some serializing code based on that. Here are some examples of what I've tried:

from typing import NamedTuple


class Base1:
    def foo(self):
        print(self.__annotations__)


class Test1(NamedTuple, Base1):
    x: int
    y: int


x = Test1(1, 2)
x.foo() # raises AttributeError


class Base2(NamedTuple):
    def foo(self):
        print(self.__annotations__)


class Test2(Base2):
    x: int
    y: int


x = Test2(1, 2) # TypeError: __new__() takes 1 positional argument but 3 were given

is there a way for me to use the NamedTuple class like this?

Solution

At issue is the metaclass used by typing.NamedTuple; this metaclass ignores all base classes and just generates a collections.namedtuple() class with added annotation information (copying across any additional attributes directly defined on the class).

You can define your own metaclass (which must be a subclass of typing.NamedTupleMeta), that adds your additional base classes after generating the named tuple class:

import typing

NamedTuple = typing.NamedTuple
if hasattr(typing.NamedTuple, '__mro_entries__'):
    # Python 3.9 fixed and broke multiple inheritance in a different way
    # see https://github.com/python/cpython/issues/88089
    NamedTuple = typing._NamedTuple

class MultipleInheritanceNamedTupleMeta(typing.NamedTupleMeta):
    def __new__(mcls, typename, bases, ns):
        if NamedTuple in bases:
            base = super().__new__(mcls, '_base_' + typename, bases, ns)
            bases = (base, *(b for b in bases if not isinstance(b, NamedTuple)))
        return super(typing.NamedTupleMeta, mcls).__new__(mcls, typename, bases, ns)

class Base1(metaclass=MultipleInheritanceNamedTupleMeta):
    def foo(self):
        print(self.__annotations__)

class Test1(NamedTuple, Base1):
    x: int
    y: int

Note that this won't let you inherit fields! That's because you must generate a new namedtuple class for any combination of fields. The above produces the following structure:

• Test1, inherits from
  • _base_Test1 - the actual typing.NamedTuple generated namedtuple
    • tuple
  • Base1

and this works as required:

>>> x = Test1(1, 2)
>>> x.foo()
{'x': <class 'int'>, 'y': <class 'int'>}

Technically speaking, you’ll only need the above in Python versions up to Python 3.10. Python 3.9 has a refactored NamedTuple implementation that fixes the specific metaclass issue underlying this problem, but then introduced a different problem by explicitly raising a TypeError stating that multiple inheritance is not supported. This TypeError was raised as a bug, as there are valid use-cases for using multiple inheritance with NamedTuple, and the exception was removed in Python 3.11.

You can safely use the code in this answer in Python 3.11, however.

Answered By - Martijn Pieters