[FIXED] Why does my customized exception_handler for Python asyncio not work as expected?

Issue

I'm relatively new to Python asyncio.

In the following code (only to put as an example) I want to use an exception handler to catch exception:

import asyncio

async def wait_with_fail(delay, fail=False):
    """
    Async function; subject to fail
    """
    while True:
        await asyncio.sleep(delay)
        if fail:
            print("wait_with_fail:", delay, " failing now...")
            1/0
        print("wait_with_fail:", delay)



def custom_exception_handler(loop, context):
    """
    Exception handler
    """
    exception = context.get('exception')
    loop.default_exception_handler(context)
    if isinstance(exception, ZeroDivisionError):
        print("In exception handler.")

def start():
    """
    Start tasks
    """
    my_loop = asyncio.get_event_loop()

    # Set custom handler
    my_loop.set_exception_handler(custom_exception_handler)

    my_loop.create_task(wait_with_fail(1))
    my_loop.create_task(wait_with_fail(10, fail=True))
    my_loop.create_task(wait_with_fail(20))

    my_loop.run_until_complete(asyncio.sleep(1000))

    print("loop finished")

if __name__ == "__main__":

    start()

It works (beside Message: Task exception was never retrieved...but this I want to understand next) - the handler is called:

wait_with_fail: 1
wait_with_fail: 1
wait_with_fail: 1
wait_with_fail: 1
wait_with_fail: 1
wait_with_fail: 1
wait_with_fail: 1
wait_with_fail: 1
wait_with_fail: 1
wait_with_fail: 10  failing now...
Task exception was never retrieved
future: <Task finished name='Task-2' coro=<wait_with_fail() done, defined at d:\HES-AT\LegacyAdapter\LegacyAdapter\src\test_q.py:3> exception=ZeroDivisionError('division by zero')>
Traceback (most recent call last):
  File "d:\HES-AT\LegacyAdapter\LegacyAdapter\src\test_q.py", line 11, in wait_with_fail
    1/0
ZeroDivisionError: division by zero
In exception handler.
wait_with_fail: 1
wait_with_fail: 1
wait_with_fail: 1
wait_with_fail: 1
wait_with_fail: 1
wait_with_fail: 1
wait_with_fail: 1
wait_with_fail: 1
wait_with_fail: 1
wait_with_fail: 1
wait_with_fail: 20
wait_with_fail: 1
wait_with_fail: 1
wait_with_fail: 1

But when I assign create_task to a variable like

...
my_task = my_loop.create_task(wait_with_fail(10, fail=True))
...

the handler is not called. I find this behavior very surprising...why does it make a difference? I observed, that the handler is not called either, when I use asyncio.gather() to start all tasks jointly.

Background: I want to have the started task in variables in order to task.cancel() them from the event loop after it has been stopped. But this is not the topic of this question and subject to another question on its own.

Solution

What seems to be happening is that the exception will only be raised when the task has no further references. Which explains why you see the exception printed but never propagated. When an exception is not retrieved, it gets handled by default by the call_exception_handler(). You can find more discussion about this topic in this stackoverflow question here.

One way you can test the reference (for example) is by replacing your start function with the following (removing the reference):

def start():
    my_loop = asyncio.get_event_loop()

    # Set custom handler
    my_loop.set_exception_handler(custom_exception_handler)

    my_loop.create_task(wait_with_fail(1))
    my_res = my_loop.create_task(wait_with_fail(10, fail=True))
    del my_res # removes reference and raises exception

    my_loop.create_task(wait_with_fail(20))

    my_loop.run_until_complete(asyncio.sleep(1000))

    print("loop finished")

As for your confusion regarding gather, it is worth mentioning that you can configure the behaviour of gather with regards to handling exception by the return_exceptions parameter. Let's write a different version of your start function as an example, which will raise the exception.

def start():
    my_loop = asyncio.get_event_loop()

    # Set custom handler
    my_loop.set_exception_handler(custom_exception_handler)

    a = my_loop.create_task(wait_with_fail(1))
    b = my_loop.create_task(wait_with_fail(10, fail=True))

    c = my_loop.create_task(wait_with_fail(20))

    tasks = asyncio.gather(a, b, c, loop=my_loop, return_exceptions=False) # here we modify how exceptions are handled

    my_loop.run_until_complete(tasks)

    print("loop finished")

On the contrary, changing the value of return_exceptions to return_exceptions=False will avoid propagating the exception. In other words (as mentioned by Dano's answer):

If return_exceptions is True, exceptions in the tasks are treated the same as successful results, and gathered in the result list; otherwise, the first raised exception will be immediately propagated to the returned future.

You can find further discussion of that issue on this stackoverflow question here.

Answered By - alt-f4