[FIXED] Gathering results from an asyncio task list with exceptions

Issue

The code...

import asyncio
import random
from time import perf_counter
from typing import Iterable
from pprint import pprint

async def coro(n, i, threshold=0.4):
    await asyncio.sleep(i)
    if i > threshold:
        # For illustration's sake - some coroutines may raise,
        # and we want to accomodate that and just test for exception
        # instances in the results of asyncio.gather(return_exceptions=True)
        raise Exception(f"{i} of Task-{n} is too high")
    return i

async def main(it: Iterable, timeout: float) -> tuple:
    tasks = [asyncio.create_task(coro(i+1, d), name=f"Task-{i+1}") for i, d in enumerate(it)]
    await asyncio.wait(tasks, timeout=timeout)
    return tasks  # *not* (done, pending)

timeout = 0.5
random.seed(444)
n = 10
it = [random.random() for _ in range(n)]
start = perf_counter()
tasks = asyncio.run(main(it=it, timeout=timeout))
elapsed = perf_counter() - start
print(f"Done main({n}) in {elapsed:0.2f} seconds\n")
pprint(tasks)
print('----')

# does not work from here on....

res = []
for t in tasks:
    try:
        r = t.result() # gives an error!!!
    except Exception as e:
        res.append(e)
    else:
        res.append(r)
pprint(res)

...does not work for collection of the task results. It fails with ...

Traceback (most recent call last):
  File "c:\Users\user\Documents\user\projects\learn\asyncio\wrap_gather_in_timeout.py", line 8, in coro
    await asyncio.sleep(i)
  File "C:\Users\user\AppData\Local\Programs\Python\Python39\lib\asyncio\tasks.py", line 654, in sleep
    return await future
asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "c:\Users\user\Documents\user\projects\learn\asyncio\wrap_gather_in_timeout.py", line 35, in <module>
    r = t.result()
asyncio.exceptions.CancelledError
Task exception was never retrieved
future: <Task finished name='Task-7' coro=<coro() done, defined at c:\Users\user\Documents\user\projects\learn\asyncio\wrap_gather_in_timeout.py:7> exception=Exception('i too high')>
Traceback (most recent call last):
  File "c:\Users\user\Documents\user\projects\learn\asyncio\wrap_gather_in_timeout.py", line 13, in coro
    raise Exception("i too high")
Exception: i too high

The code was run in python 3.9. Any idea where I am going wrong and why?

Is it because the tasks need to be cancelled after it throws an exception? I could not successfully implement it.

Inspired by: Solution to wrapping asyncio.gather SO

Solution

Your code works, the issue why you are not able to create res successfully is because the code does not raise just the normal Exception class. Since the task fails it ends up calling asyncio.exceptions.CancelledError which if we take a look in the documentation inherits from BaseException not Exception. This change is new as of Python 3.8 and since you are using Python 3.9 that change is live. Changing your code slightly to the following yields:

res = []
for t in tasks:
    try:
        r = t.result() # gives an error!!!
    except BaseException as e:
        res.append(e)
        continue
    res.append(r)

print(res)
[0.3088946587429545,
 0.01323751590501987,
 Exception('0.4844375347808497 of Task-3 is too high'),
 asyncio.exceptions.CancelledError(),
 asyncio.exceptions.CancelledError(),
 asyncio.exceptions.CancelledError(),
 Exception('0.4419557492849159 of Task-7 is too high'),
 0.3113884366691503,
 0.07422124156714727,
 asyncio.exceptions.CancelledError()]

Answered By - gold_cy