[FIXED] python asyncio run_in_executor never done

Issue

I running the following code in python 3.10.12. There is a sync function fetch_data, which should return 1 immediately. And in main function, a CPU-bound task like sum(i for i in range(int(1e6))) which cost 4~5s.

import asyncio

def fetch_data():
    print("done")
    return 1

async def main():
    loop = asyncio.get_event_loop()
    task = loop.run_in_executor(None, fetch_data)
    sum(i for i in range(int(1e6)))
    print(task.done())

asyncio.run(main())

The output I expected is

done
True

But what I got is

done
False

The fetch_data seems never done, but when I insert await asyncio.sleep(0) before print(task.done()), the output is what I expected.

import asyncio

def fetch_data():
    print("done")
    return 1

async def main():
    loop = asyncio.get_event_loop()
    task = loop.run_in_executor(None, fetch_data)
    sum(i for i in range(int(1e6)))
    await asyncio.sleep(0) # After sleep(0), the output is as expected.
    print(task.done())

asyncio.run(main())

Why is this happening? What should I do to achieve what I needed without asyncio.sleep(0)?

Solution

It's happening because you don't give the executor a change to process it's tasks. With await asyncio.sleep(0) you interrupt your main() function and executor process the tasks and the result is as expected.

If asyncio.sleep is problem, you can use other executor:

import asyncio
import concurrent.futures


def fetch_data():
    print("done", flush=True)
    print()
    return 1


async def main():
    with concurrent.futures.ThreadPoolExecutor(max_workers=2) as executor:
        task = executor.submit(fetch_data)
        sum(i for i in range(int(1e8)))
        print(task.done())


asyncio.run(main())

Prints:

done

True

Answered By - Andrej Kesely