[FIXED] Unexpected DeprecationWarning in generator when using IPython

Issue

I am using a modified form of the grouper recipe provided by the python docs:

from itertools import chain, islice
def grouper(iterable, n):
    iterable = iter(iterable)
    while True:
        peek = next(iterable)
        yield chain((peek,), islice(iterable, n - 1))

This seems to work fine. I can do something like:

>>> x = bytearray(range(16))
>>> [int.from_bytes(n, 'big') for n in grouper(x, 4)]
[66051, 67438087, 134810123, 202182159]

However, when I run the exact same code in IPython, I get a DeprecationWarning:

In [1]: from itertools import chain, islice
   ...: def grouper(iterable, n):
   ...:      iterable = iter(iterable)
   ...:      while True:
   ...:          peek = next(iterable)
   ...:          yield chain((peek,), islice(iterable, n - 1))

In [2]: x = bytearray(range(16))

In [3]: [int.from_bytes(n, 'big') for n in grouper(x, 4)]
__main__:1: DeprecationWarning: generator 'grouper' raised StopIteration
Out[3]: [66051, 67438087, 134810123, 202182159]

Where is the warning coming from and why do I not see it in the regular Python console? What can I do to make the warning go away?

I am using Python 3.6.2 and IPython 6.1.0

Solution

This is a change to Python that is being gradually phased in between Python 3.5 and Python 3.7. The details are explained in PEP 479, but I'll try to give a quick overview.

The issue is StopIteration exceptions that are leaking out of generator functions. It might seem like that's OK, since raising a StopIteration is the signal an iterator is finished. But it can cause problems with refactoring generators. Here's an example showing the trouble:

Say you had this generator function (which works fine in Python versions prior to 3.5, where it started emitting warnings):

def gen():
    yield 1
    yield 2
    if True:
        raise StopIteration
    yield 3
    yield 4

Since the condition of the if is truthy, the generator will stop after yielding two values (not yielding 3 or 4). However, what if you try to refactor the middle part of the function? If you move the part from yield 2 to yield 3 into a helper generator, you'll see an issue:

def gen():
    yield 1
    yield from refactored_helper()
    yield 4

def refactored_helper():
    yield 2
    if True:
        raise StopIteration
    yield 3

In this version, 3 will be skipped, but 4 will still be yielded. That's because the yield from ate the StopIteration that was raised in the helper generator function. It assumed that only the helper generator should be stopped, and so the outer generator kept running.

To fix this, the Python devs decided to change how generators work. Starting in Python 3.7, a StopIteration exception that leaks out of a generator function will be changed by the interpreter to a RuntimeError exception. If you want to exit a generator normally, you need to use return. Furthermore, you can now return a value from a generator function. The value will be contained in the StopIteration exception that gets raised by the generator machinery, and a yield from expression will evaluate to the returned value.

So the generators above could properly be refactored to:

def gen():
    yield 1
    if yield from refactored_helper():
        return
    yield 4

def refactored_helper():
    yield 2
    if True:
        return True
    yield 3
    # like a normal function, a generator returns None if it reaches the end of the code

If you want to write future-compatible code now, you should put from __future__ import generator_stop at the top of your module. Then you need to track down places where you're leaking StopIteration exceptions and wrap them with try and except logic. For the code in your question:

from __future__ import generator_stop

from itertools import chain, islice

def grouper(iterable, n):
    iterable = iter(iterable)
    while True:
        try:
            peek = next(iterable)
        except StopIteration:
            return
        yield chain((peek,), islice(iterable, n - 1))

Answered By - Blckknght