"Retry" from tenacity module doesn't work with a generator

Question

I have an issue using the "retry" tool from the tenacity library in python3. The "retry" decorator does not seem to work when I use a generator.

I have a code sample to illustrate my situation:

from tenacity import retry, wait_exponential

@retry(wait=wait_exponential(multiplier=1, min=1, max=1))
def test_retry():
print("test retry from tenacity")
for i in range(10):
    if i == 0: raise Exception
    yield i

def gen():
    yield from test_retry()

bar = gen()
for foo in bar:
    print(foo)

When it raises the Exception, it doesn't retry. Does somebody have an idea why this is not working?

Thanks

Answer 1

It's a bug/feature/quagmire with Tenacity itself where retry logic fails on generator functions. Tenacity devs state that this is because "generators use exceptions internally." The original developer further goes on the write that "tenacity.retry() wrapped the generator function, not the generator itself (i.e. the user's code)." Basically there doesn't look to be any plans to work around this behavior, if it can even be done.

To deal with it, Tenacity annotations should be added to methods calling the generators -- where Exceptions can be caught readily by Tenacity as they bubble up through the call stack. Important: The generator functions must not hide exceptions, either.

# in generator do this and add retry annotations to calling method
...
try: 
    do_something()
except Exception as ex: 
    log_or_do_something_else()
    raise
finally: 
    cleanup()
yield something
...


# in generator don't do this
...
try: 
    do_something()
except Exception as ex: 
    pass
yield something
...