[FIXED] python: iterate over either a list or an async generator

Issue

Since iterators were introduced in python, it's always been possible to not care whether you are dealing with an iterator or a list:

from random import random

def gen_list():
    print('gen')
    for i in range(10):
        yield i

def return_list():
    print('return')
    return [i for i in range(10)]


if random() > 0.5:
    x = gen_list()
else:
    x = return_list()

for i in x:
    pass

PEP 492 introduced asynchronous iterators and the async for syntax. What I can't see is any justification for the new burden of adding syntax to the consumer of the async iterator.

In my code, I sometimes am dealing with a list (from a cache), and sometimes with an async generator:

import asyncio
from random import random

def is_small_and_in_cache():
    if random() > 0.5:
        print('in fake cache')
        return [i for i in range(10)]

async def get_progressively():
    print('gen')
    for i in range(10):
        # e.g. an await here
        await asyncio.sleep(0.1)
        yield i

async def main():
    x = is_small_and_in_cache()
    if x is None:
        x = get_progressively()

    async for i in x:
        pass

asyncio.run(main())

But the above fails (half the time) with TypeError: 'async for' requires an object with __aiter__ method, got list.

Main Question: How to write this so that we can deal with either? Should I try to convert the list to a dummy async generator, or wrap the async generator so that it produces a list?

Side Quest: Are there any proposals to get rid of the (clearly unpythonic, to me) async for construct, i.e. why can't a regular for loop handle an asynchronous generator? Has Python3x lost it's way in terms of usability??

Solution

The syntax exists to warn you that your “loop” might actually include suspending your entire call, allowing other code to run, so that you know to have appropriate data in a consistent state at the top of each iteration. It’s not going anywhere.

Of course, a coroutine doesn’t have to suspend, and you can use that to make wrapping any iterable trivial:

async def desync(it):
  for x in it: yield x

This is more generally useful than the opposite number which—still asynchronously, as it must—gathers into a list:

async def gather(ai):
  ret=[]
  async for x in ai: ret.append(x)
  return ret

since it allows for proper interleaving in the fully asynchronous case.

Answered By - Davis Herring