[FIXED] Why gather() is faster than for loop in asyncio?

Issue

Why first code is slower than second? (Only the definition of ret has changed) This is a fragment of my code and in the full program, about fifty links are fed into the main function.

async def get(url, session):
    try:
        async with session.get(url) as response:
            resp = await response.read()
            print(f"Successfully got url {url} with resp of length {len(resp)}.")
    except Exception as e:
        print("Unable to get url {} due to {}.".format(url, e.__class__))


async def main(urls):
    session = aiohttp.ClientSession()
    ret = await asyncio.gather(*[get(url, session) for url in urls])
    print("Finalized all. Return is a list of len {} outputs.".format(len(ret)))
    await session.close()

Second:

async def get(url, session):
    try:
        async with session.get(url) as response:
            resp = await response.read()
            print(f"Successfully got url {url} with resp of length {len(resp)}.")
    except Exception as e:
        print("Unable to get url {} due to {}.".format(url, e.__class__))


async def main(urls):
    session = aiohttp.ClientSession()
    ret = [await get(url, session) for url in urls]
    print("Finalized all. Return is a list of len {} outputs.".format(len(ret)))
    await session.close()

The first code runs for 1.5 seconds, and the second for 8-10 seconds

Solution

In both cases, you have a list comprehension that creates a list of ... something using a list of URLs.

In the first case, you create a list of coroutines, but don't execute them immediately. Instead, you supply all of them to gather, which executes them concurrently, completing once all the coroutines have completed.

In the second case, you create a list of None values, but each coroutine created by get has to be executed to completion before the next coroutine can be created. Using await schedules the coroutine for execution and allows main to suspend while it is blocked waiting on the coroutine to complete.

Answered By - chepner