[FIXED] IPython REPL error when using a custom __getattribute__

Issue

I have a custom __getattribute__ that is supposed to modify the returned value if the member is not a method (thus an attribute). Assume all the attributes (self.a, self.b, etc) are str.

class A:
    def __init__(self):
        self.a = 1

    def __getattribute__(self, k):
        attr = object.__getattribute__(self, k)
        if type(attr) != types.MethodType:
            return '{}!'.format(attr)
        return attr

I get an error in IPython when getting the representation of instances of class A but I don't understand why.
For example:

In [26]: a = A()
In [27]: a
Out[27]: ---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
~/miniconda3/lib/python3.7/site-packages/IPython/core/formatters.py in __call__(self, obj)
    700                 type_pprinters=self.type_printers,
    701                 deferred_pprinters=self.deferred_printers)
--> 702             printer.pretty(obj)
    703             printer.flush()
    704             return stream.getvalue()

~/miniconda3/lib/python3.7/site-packages/IPython/lib/pretty.py in pretty(self, obj)
    380             #   1) a registered printer
    381             #   2) a _repr_pretty_ method
--> 382             for cls in _get_mro(obj_class):
    383                 if cls in self.type_pprinters:
    384                     # printer registered in self.type_pprinters

~/miniconda3/lib/python3.7/site-packages/IPython/lib/pretty.py in _get_mro(obj_class)
    318         # Old-style class. Mix in object to make a fake new-style class.
    319         try:
--> 320             obj_class = type(obj_class.__name__, (obj_class, object), {})
    321         except TypeError:
    322             # Old-style extension type that does not descend from object.

AttributeError: 'str' object has no attribute '__name__'

But print(a) works fine

In [33]: print(a)
<__main__.A object at 0x10c566390>

Note: In the plain Python REPL it seems to be working properly.

>>> a = A()
>>> a
<__main__.A object at 0x1032b9320>

Solution

In IPython, the standard output displays a pretty printed __repr__ representation of the object. While in Python, the standard output prints the __repr__ representation of the object, in short, print(repr(obj)).

Python:

As you will notice below, the standard output in Python is the same as calling the print() function on the repr(a). repr(a) is the object representation of a and invokes __repr__ when called.

>>> a = A()
>>> a
<__main__.A object at 0x000000D886391438>
>>> repr(a)
'<__main__.A object at 0x000000D886391438>'
>>> print(repr(a))
<__main__.A object at 0x000000D886391438>

IPython:

IPython, on the other hand, has its own implementation of displaying the standard output and pretty prints the __repr__ for the object before displaying. The pretty printing of the object for stdout happens in the pretty() function located in the RepresentationPrinter class in ../IPython/lib/pretty.py:

def pretty(self, obj):
        """Pretty print the given object."""
        obj_id = id(obj)
        cycle = obj_id in self.stack
        self.stack.append(obj_id)
        self.begin_group()
        try:
            obj_class = _safe_getattr(obj, '__class__', None) or type(obj)
        #<---code--->

However, before pretty() is called, IPython calls the __call__(self,obj) method in ../IPython/core/formatters.py. You will notice this as the topmost stack in the Traceback Exception error and the pretty() function above is called on line 702:

AttributeError                            Traceback (most recent call last)
~/miniconda3/lib/python3.7/site-packages/IPython/core/formatters.py in __call__(self, obj)
    700                 type_pprinters=self.type_printers,
    701                 deferred_pprinters=self.deferred_printers)
--> 702             printer.pretty(obj)

In the pretty() function above the _safe_getattr(obj, '__class__', None) or type(obj) line is interesting. The definition for this function says it is a safe implementation of getarr() which means that if an Exception is raised while getting the attribute for this object, it will return None:

def _safe_getattr(obj, attr, default=None):
"""Safe version of getattr.

Same as getattr, but will return ``default`` on any Exception,
rather than raising.
"""
    try:
        return getattr(obj, attr, default)
    except Exception:
        return default

In the pretty() function, the value of _safe_getattr(obj, '__class__', None) or type(obj) is stored in obj_class. Later, in the same function, this variable is passed to _get_mro(). This is shown in the second stack of the Traceback Exception on line 382:

    ~/miniconda3/lib/python3.7/site-packages/IPython/lib/pretty.py in pretty(self, obj)
    380             #   1) a registered printer
    381             #   2) a _repr_pretty_ method
--> 382             for cls in _get_mro(obj_class):
    383                 if cls in self.type_pprinters:
    384                     # printer registered in self.type_pprinters

The job of _get_mro(obj_class) is to get an MRO(method resolution order) for obj_class. In Python 3, all classes are new style and have a __mro__ attribute. However, old style class definitions have been retained for backward compatibility and do not have this attribute. Your class is defined with the old-style syntax. You can read more about NewClass v/s OldClass here. In the definition for _get_mro(obj_class), your code falls into the try block for old-style syntax and errors out. This is the latest and the bottommost stack in the Traceback Exception:

  ~/miniconda3/lib/python3.7/site-packages/IPython/lib/pretty.py in _get_mro(obj_class)
    318         # Old-style class. Mix in object to make a fake new-style class.
    319         try:
--> 320             obj_class = type(obj_class.__name__, (obj_class, object), {})
    321         except TypeError:

So what's happening:

Let us use all that we have learned and understand what is really happening behind the scenes. I've modified your code below to make use of the above functions from the IPython module. You should try this on the IPython console/Jupyter notebook:

    In [1]: from IPython.lib.pretty import _safe_getattr
       ...: from IPython.lib.pretty import pretty
       ...: from IPython.lib.pretty import _get_mro
       ...:
       ...: class A:
       ...:     def __init__(self):
       ...:         self.a = 1
       ...:
       ...:     def __getattribute__(self, k):
       ...:         attr = object.__getattribute__(self, k)
       ...:         if type(attr) != types.MethodType:
       ...:             return '{}!'.format(attr)
       ...:         return attr
       ...:
       ...: a = A()
       ...: a.test_attr = 'test_string'
    In [2]: getattr_res = _safe_getattr(a, 'test_attr') or type(a)
    In [6]: getattr_res
    Out[6]: 'test_string!'
    In [10]: getattr_res == getattr(a, 'test_attr')
    Out[10]: True

I've defined an attribute test_attr which stores a string 'test_string' as you've mentioned all attributes are str. The getattr_res variable stores the value for invoking _safe_getattr(a, 'test_attr') which is same as invoking getattr(a, 'test_attr') which basically calls __getattribute__ in your code:

In [13]: a.__getattribute__('test_attr')
Out[13]: 'test_string!'

As you will observe getattr_res is of type string and string objects do not have the __mro__ attribute. We should have a class object to get the MRO:

In [14]: type(getattr_res)
Out[14]: str
In [15]: _get_mro(getattr_res)
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-15-d0ae02b5a6ac> in <module>()
----> 1 _get_mro(getattr_res)

C:\ProgramData\Anaconda3\lib\site-packages\IPython\lib\pretty.py in _get_mro(obj_class)
    316         # Old-style class. Mix in object to make a fake new-style class.
    317         try:
--> 318             obj_class = type(obj_class.__name__, (obj_class, object), {})
    319         except TypeError:
    320             # Old-style extension type that does not descend from object.

AttributeError: 'str' object has no attribute '__name__'

This Exception looks familiar isn't it? The call to IPython's _safe_getattr(obj, '__class__', None) function invokes __getattribute__ in your code which returns a string object which does not have a __mro__ attribute and even if _get_mro(obj_class) attempts execution in the try block we get an AttributeError because we know that str objects do not have a '__name__' attribute:

In [16]: getattr_res.__name__
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-16-0d8ba2c5af23> in <module>()
----> 1 getattr_res.__name__

AttributeError: 'str' object has no attribute '__name__'

How do we fix this:

In IPython, it is possible to add our own pretty printing rules for the objects in our class. Inspired by the docs for module lib.pretty, I've modified the code and defined a _repr_pretty_(self, p, cycle) function that is explicitly called in __getattribute__ (after the type check) to display the object in the desired format. If the attribute is a string, it simply returns the string again:

In [20]: class A:
    ...:     def __init__(self):
    ...:         self.a = 1
    ...:
    ...:     def __getattribute__(self, k):
    ...:         attr = object.__getattribute__(self, k)
    ...:         if type(attr) != types.MethodType:
    ...:             return self._repr_pretty_(attr, cycle=False)
    ...:         return attr
    ...:
    ...:     def _repr_pretty_(self, p, cycle):
    ...:         if cycle:
    ...:             p.text('MyList(...)')
    ...:         else:
    ...:             if isinstance(p,str):
    ...:                 return p
    ...:             return p.text(repr(self) + '!')

In [21]: a = A()
In [22]: a
Out[22]: <__main__.A object at 0x0000005E6C6C00B8>!
In [24]: a.test = 'test_string'
In [25]: a.test
Out[25]: 'test_string'

Note that cycle=False when calling _repr_pretty_() in __getattribute__(self, k) because attr is not an iterable.

In general, it is recommended to add a __repr__ function to your class as it clearly shows the representation of objects in your class. You can read more about it here.

Conclusion: IPython standard output implements its own pretty printed __repr__ as opposed to the Python interpreter that utilizes the built-in repr() function for stdout. In order to change the behavior of the stdout on IPython, one can add a _repr_pretty_() function to their class to display output as desired.

Answered By - amanb