[FIXED] Cancelling a Task in asyncio internals

Issue

I know that task.cancel() doesn't do anything except setting a flag which says the task should be cancelled(here), so that in the next iteration of the event loop, a CancelledError gets thrown into the wrapped coroutine.

Also I know that throwing exceptions into coroutines only happens at await statement calls. The coroutine has to be in that state.

Now take a look at this:

import asyncio

async def coro_1():
    print("inside coro_1")
    try:
        await asyncio.sleep(0)
    except asyncio.CancelledError:
        print("cancellation error occurred")
        raise

async def main():
    print("inside main")
    new_task = asyncio.create_task(coro_1())
    print("finishing main")

asyncio.run(main())

output:

inside main
finishing main
inside coro_1
cancellation error occurred

In main() coroutine, I created a new task, but I didn't give it opportunity to run. (I didn't await on anything). The main() coroutine finishes but the coro_1 executes!

Ok I know that this happened because I used asyncio.run() but after I went through the source code, I couldn't understand why and where. I must be definitely wrong in one of the following steps.

1. when coro_1() task is created in main, it schedules a new callback(self.__step) in the event loop for the next cycle. (in self._ready queue).

2. the main is finished, execution goes into the __exit__ method of the Runner object(here).

3. it then .cancel()s all remaining tasks here. Now we have our coro_1 task. It's _must_cancel flag is set and its __step is waiting to be executed in the next iteration with that exception set.

4. Since the exc is set, we reach at this line:

   ...
   else:
       result = coro.throw(exc)
   ...
   

Now how is that possible? our coro_1 is not in the state that we can throw exception to it. Someone must have executed it until the line await asyncio.sleep(0) in the past. who and where?

I use Python 3.11.

Thanks in advance.

Solution

I found the answer.

First off, there is a queue called self._ready() in the event loop within which all callbacks are ready to be executed.

These are the steps:

1. When we run our main coroutine using .run(), it creates a new task, add its callback to the self._ready, then it calls run_until_complete(). Our self._ready only contains that callback.

2. In run_until_complete, it adds _run_until_complete_cb callback to the add_done_callback of the future.(it's gonna be called the future is done). what is its job? it stops the event loop. it basically sets self._stopping attribute of the event loop to True, which is used here.

3. Now it calls run_forever() which indeed iterates through self._ready queue and runs all callbacks. (right now we have only one. the main coroutine).

4. Note that callbacks are popped out of the self._ready in order to be called, so the self._ready is empty when execution reaches the line new_task = asyncio.create_task(coro_1()). It then adds new callback to the self._ready which is responsible for running coro_1 coroutine.

5. the main() reaches the end and it's finished. what happens next? it's time for scheduling its _run_until_complete_cb which we registered earlier.

6. Now what is inside our self._ready ? two callbacks, one for executing coro_1 and one for executing _run_until_complete_cb. The loop is not stopped yet. So another cycle of the loop is fired. It executes coro_1 until the await statement(it's now ready for CancelledError exception), and also executes _run_until_complete_cb. This latter callback, stops the eventloop. what is inside our self._ready? only one callback which is responsible for continuing the coro_1. (it scheduled itself back to the eventloop).

7. Now it's time for __exit__ method of Runner object to be called. what does it do? it cancels all remaining tasks. our only task is coro_1 which is pending. Now in the third cycle of the event loop it throws CancelledError into coro_1.

Answered By - S.B