[FIXED] Call asyncio Future in scope

Issue

I'm trying to get the progession of my asyncIO Futures. For some reason, I can't get value updates inside any other scope.

For example:

import concurrent.futures
import time
import asyncio
import random

def get_progress(futures):
    return sum([f.done() for f in futures])

def long_task(t):
    time.sleep(1.5)
    return t

loop = asyncio.get_event_loop()
executor = concurrent.futures.ProcessPoolExecutor(max_workers=4)
inputs = ['a', 'b', 'c', 'd', 'e', 'f', 'g']
futures_ = [loop.run_in_executor(executor, long_task, i) for i in inputs]

for i in range(5):
    time.sleep(1)
    print(get_progress(futures_))

It prints only 0. However, if I run this inside a terminal and call get_progress(futures_) it prints 7 as expected.

Am I missing something here ?

Solution

Asyncio futures returned by run_in_executor are designed to run in the asyncio event loop and are distinct from concurrent ones. The correct asyncio code would look like this:

async def track_progress():
    for i in range(5):
        await asyncio.sleep(1)
        print(get_progress(futures_))

loop.run_until_complete(track_progress())

It prints 0 4 4 7 7.

Note that you don't need asyncio and run_in_executor if you just want to run stuff in threads - concurrent.futures is perfectly capable of handling that on its own. For example:

executor = concurrent.futures.ProcessPoolExecutor(max_workers=4)
inputs = ['a', 'b', 'c', 'd', 'e', 'f', 'g']
futures_ = [executor.submit(long_task, i) for i in inputs]

Since executor.submit real concurrent futures, futures_ contains them as well. With this modification your original for loop that tracks their progress will work just fine and also print 0 4 4 7 7.

Answered By - user4815162342