[FIXED] (Python) How to run tasks concurrently (and independently) without using asyncio gather?

Issue

I have a pool of tasks in a list and each of those tasks take a different amount of time to complete. To mimick this I'll use this piece of code: tasks = [asyncio.create_task(asyncio.sleep(i)) for i in range(10)]

If I use the asyncio gather API like this: await asyncio.gather(*tasks), then this statement blocks the event loop untill all the tasks in the current batch finishes.


The problem however is in my program, I want to execute those tasks with a ThreadPoolExecutor like interface with x number of workers. This will allow for executing tasks more efficiently (Tasks which finish up early will be replaced by another task instead of waiting for all tasks of that batch to complete).

How can I achieve this using python asyncio?

Solution

asyncio.gather() waits for the tasks to finish. If you don’t want to wait, just don’t call asyncio.gather().

The tasks will kick off even if you don’t call gather(), as long as your event loop keeps running.

To keep an event loop running, call loop.run_forever() as your entry point, or call asyncio.run(coro()) and in coro() call await asyncio.Future() to “wait forever”.

You may also check out the TaskGroup class of Python 3.11 to see if it’s semantics meet what you need.

Answered By - fancidev