[FIXED] Unexpected behavior from the "yield from/await" statement?

Issue

Consider the following asyncio code:

import asyncio

class A:
    async def noop(self):
        pass
    async def f(self, name):
        print(f"{name} -> await")
        await self.noop()
        print(f"{name} -> return")
    async def f1(self):
        await self.f("f1")
    async def f2(self):
        await self.f("f2")
    async def f3(self):
        await self.f("f3")
    async def __call__(self):
        await asyncio.gather(self.f1(), self.f2(), self.f3())

class B(A):
    async def noop(self):
        await asyncio.sleep(0)

async def main():
    a = A()
    print(a.__class__.__name__)
    await a()
    print()
    b = B()
    print(b.__class__.__name__)
    await b()

asyncio.run(main())

Output:

A
f1 -> await
f1 -> return
f2 -> await
f2 -> return
f3 -> await
f3 -> return

B
f1 -> await
f2 -> await
f3 -> await
f1 -> return
f2 -> return
f3 -> return

A quick look at asyncio sleep's code gives us this:

@types.coroutine
def __sleep0():
    """Skip one event loop run cycle.
    This is a private helper for 'asyncio.sleep()', used
    when the 'delay' is set to 0.  It uses a bare 'yield'
    expression (which Task.__step knows how to handle)
    instead of creating a Future object.
    """
    yield


async def sleep(delay, result=None):
    ...
    if delay <= 0:
        await __sleep0()
        return result
    ...

And a quick look at asyncio Task __step's code shows that it can only return control to the event loop if there is a yielded object and it's either None or a future-like object implementing the Future's class interface, namely a _asyncio_future_blocking attribute and a similar __await__ method.

Question:

The way the asyncio docs and examples are worded give the impression that a coroutine is always suspended when an await expression is run.

Since await obj is equivalent to yield from obj.__await__() and yield generally means "yield control/suspend", was it a oversight that yield from won't suspend generators if nothing was yielded?

def outer():
    def inner(v):
        if v: yield
    yield from inner(False) # Not suspended
    yield from inner(True)  # Suspended

I understand that it wouldn't be feasible to suspend a non-yielding generator with yield from, but the fact that it's named yield and silently goes through (no StopIteration exception) caused quite a confusion in my mind and my code, when it wasn't suspending a task mid-execution.

Asyncio sleep's docs mention it without detailing the trick involved.

Setting the delay to 0 provides an optimized path to allow other tasks to run. This can be used by long-running functions to avoid blocking the event loop for the full duration of the function call.

Solution

Your exact question was:

"was it a[n] oversight that yield from won't suspend generators if nothing was yielded?"

The answer is that it absolutely, positively was not an oversight. There is no room for any doubt. Consider the following code, where g is a generator/iterator:

next(g)

or the equivalent:

g.__next__()

When you call this function, one of two things must happen. It either returns an object, possibly None; or it raises an exception. That is true of any Python function call. It's how the language works.

So if "nothing was yielded" by g, there is no possible definition for the next function. Returning None is a perfectly valid possibility, but "returning nothing" is not.

Consider:

def f1():
    yield None

def f2():
    if False:
         yield None

x1 = list(f1())
x2 = list(f2())

print(x1)
>>> [None]
print(x2)
>>> []

The first iterator yields one object, None. The second iterator never yields at all. Both are valid but they are not the same thing.

Thus there is no such concept as "suspending an generator/iterator if nothing was yielded." All yields must yield some "thing" - some object. That object is the returned value of the next function. It is possible to design a programming language that works some other way, but it wouldn't be Python.

As for StopIteration, it is only raised when the iterator/generator returns. When you use an iterator in an in expression, Python suppresses this error. The only way I can think of to see an explicit StopIteration is to call one of the generator's low-level functions, like next or send. Your expectation of seeing StopIteration from code like yield from x, where x yields nothing, is not correct. This code:

x = []
yield from x

does not terminate a function, it's just a no-op. It won't raise anything and the method will continue.

I hope this clarifies some of these issues.

Answered By - Paul Cornelius