[FIXED] Add coroutine to gather array

Issue

I have an array of coroutines in asyncio, all of the are blocking, and in one of them would like to add a new one to the array to be gathered. Not sure if this is possible ... Imagine the pingLoop code needs to add a new routine pingSpecial() and await it, it will be also blocking, so it would block the pingLoop, which is not OK

loops=[pingLoop()]
await asyncio.gather(*loops)

Solution

I think I got what you mean despite of using the wrong "blocking" word. Coroutine shouldn't be blocking otherwise no other coroutine will have the opportunity to run in between. So they will be executed sequentially and you won't benefit from asynchronous programming.

asyncio.gather() is a useful feature when you want to await on multiple coroutines(which should be of course non-blocking). But it has some draw backs. One of them is that "you cannot add another coroutine to the list"!

This is just to prove:

import asyncio

async def newly_added_coro():
    print("newly added coro starts")
    await asyncio.sleep(5)
    print("newly added coro finished")

async def ping_coro(n, add_task=False):
    if add_task:
        t = asyncio.create_task(newly_added_coro())

    for _ in range(3):
        print(f"coro {n} is pinging...")
        await asyncio.sleep(1)

async def main():
    tasks = [ping_coro(1), ping_coro(2), ping_coro(3, add_task=True)]
    await asyncio.gather(*tasks)

asyncio.run(main())

Run this and you see it doesn't wait for newly_added_coro (you'll not see print("newly added coro finished") printed to the output.

Fortunately, Asyncio added TaskGroup^[1] which gives you exactly this ability(among other useful benefits).

import asyncio

async def newly_added_coro():
    print("newly added coro starts")
    await asyncio.sleep(5)
    print("newly added coro finished")

async def ping_coro(n, add_task=False, task_group=None):
    if add_task:
        task_group.create_task(newly_added_coro())

    for _ in range(3):
        print(f"coro {n} is pinging...")
        await asyncio.sleep(1)

async def main():
    async with asyncio.TaskGroup() as tg:
        tg.create_task(ping_coro(1))
        tg.create_task(ping_coro(2))
        tg.create_task(ping_coro(3, add_task=True, task_group=tg))

asyncio.run(main())

As long as you have a reference to the tg object, you can use it inside your coroutines and add another task and wait on it along with the others.

^[1]: This is added in Python 3.11

Answered By - S.B