[FIXED] Process an Asynchronous Iterator in background

Issue

I trying to process an asynchronous iterator in background.

I can process in the main function:

target = ...
async def main():
  async for item in target.listen():
    print(item) # works fine

  print('done') # never arrives, because loop iterates forever

asyncio.run(main())

But doing it this way, the code is blocking.

So I try process the iterator in other function with create_task, but the loop never iterates.

target = ...
async def process():
  async for item in target.listen():
    print(item) # never print

async def main():
  asyncio.create_task(process())
  print('done') # works

asyncio.run(main())

Is it possible to process an asynchronous iterator in background? How?

Solution

If you call create_task() and then immediately exit -- which is what you're doing in your code -- then of course the task never runs. You can only run something "in the background" if you're doing something in the foreground, and you're not.

Even if your main function wasn't exiting immediately, your background task still wouldn't run. Asyncio provides concurrent execution, but not parallel execution -- other tasks won't execute unless the current task yields control by calling await.

Here's a working example; I've replaced your target variable with a simple list of words just so we have something over which to iterate:

import asyncio

target = "one two three four five six".split()


async def process():
    for item in target:
        print(item)  # never print
        await asyncio.sleep(0.3)


async def main():
    task = asyncio.create_task(process())
    while not task.done():
        print("doing something while waiting")
        await asyncio.sleep(1)
    print("done")


asyncio.run(main())

Running the above code produces:

doing something while waiting
one
two
three
four
doing something while waiting
five
six
done

---

Rather than checking for task.done() as I've done in this example, you would in practice be more likely to use one of the waiting primitives or asyncio.gather to wait for the completion of background tasks.

Answered By - larsks