[FIXED] Why this asyncio code doesn't run concurrent?

Issue

Hello I am trying to write a script to process pdf files with asyncio concurrent in order to do this I have the following code:

import click
import asyncio
from pdf2image import convert_from_path
from functools import wraps

def coro(f):
    @wraps(f)
    def wrapper(*args, **kwargs):
        return asyncio.run(f(*args, **kwargs))

    return wrapper

async def my_coroutine(path):
    print(path)
    return convert_from_path(path, fmt="ppm", poppler_path="")

@click.command()
@click.option("-s", "settings_path", required=False, type=str):
@coro
async def dlr(settings_path) -> None:
    paths = [...]

    responses = await asyncio.gather(*[my_coroutine(path) for path in path])

@click.group()
def cli() -> None:
    pass

cli.add_command(dlr)

if __name__ == "__main__":
    cli()

When run this is running sequential instead of "parallel", how can I improve this?

Thanks

Solution

Asyncio doesn't run things in parallel; it runs them concurrently. Only a single coroutine is actually running at any time, and other coroutines will only run if the currently running coroutine calls await to yield control.

Your my_coroutine task never awaits on anything, so no other task can run until the current one is complete. Since it's only calling a single function, and that function isn't itself a coroutine, it's not clear there's really any way to make effective use of asyncio with your code.

Asyncio only makes sense for io-bound programs (tasks can yield control to other tasks when they are waiting for io). For programs that are primarily CPU-bound, other concurrently tools like the multiprocessing module may make more sense.

Answered By - larsks