[FIXED] Confused by python async for loop--executes sequentially

Issue

I am new to asyncio and trying to understand basic for loop behavior. The code below executes sequentially, but my naive assumption was that while the sleeps are occurring, other items could be fetched via the for loop and start processing. But that doesn't seem to happen.

For example, while the code is "doing something else with 1" it seems like it could fetch the next item from the loop and start working on it while waiting for the sleep to end on item 1. But when I run, it executes sequentially with pauses for the sleeps like a non-async program.

What am I missing here?

import asyncio


class CustomIterator():
    def __init__(self):
        self.counter = 0

    def __aiter__(self):
        return self

    async def __anext__(self):
        if self.counter >= 3:
            raise StopAsyncIteration
        await asyncio.sleep(1)
        self.counter += 1
        return self.counter


async def f(item):
    print(f"doing something with {item}")
    await asyncio.sleep(3)


async def f2(item):
    print(f"doing something else with {item}")
    await asyncio.sleep(2)


async def do_async_stuff():
    async for item in CustomIterator():
        print(f"got {item}")
        await f(item)
        await f2(item)


if __name__ == '__main__':
    asyncio.run(do_async_stuff())

Output:

got 1
doing something with 1
doing something else with 1
got 2
doing something with 2
doing something else with 2
got 3
doing something with 3
doing something else with 3

Solution

I think you have a common misunderstanding of how async works. You have written your program to be synchronous. await foo() says to call foo(), and feel free to go do something else while we're waiting for foo to return with its answer. Likewise, getting the next element from your custom iterator says "get the next element of this iterator, but feel free to go do something else while waiting for the result". In both cases, you have nothing else to do, so your code wants.

If it is safe for two things in your code to run at once, it is your job to say so, using appropriate primitives.

Answered By - Frank Yellin