[FIXED] How to run a blocking code independently from asyncio loop

Issue

My project requires me to run a blocking code (from another library), whilst continuing my asyncio while: true loop. The code looks something like this:

async def main():
    while True:
        session_timeout = aiohttp.ClientTimeout()
        async with aiohttp.ClientSession() as session:
            
            // Do async stuffs like session.get and so on
            
            # At a certain point, I have a blocking code that I need to execute
            
            // Blocking_code() starts here. The blocking code needs time to get the return value.
               Running blocking_code() is the last thing to do in my main() function.
            
            # My objective is to run the blocking code separately. 
            # Such that whilst the blocking_code() runs, I would like my loop to start from the beginning again,
            # and not having to wait until blocking_code() completes and returns.
            # In other words, go back to the top of the while loop.
            
            # Separately, the blocking_code() will continue to run independently, which would eventually complete
            # and returns. When it returns, nothing in main() will need the return value. Rather the returned
            # result continue to be used in blocking_code()

asyncio.run(main())

I have tried using pool = ThreadPool(processes=1) and thread = pool.apply_async(blocking_code, params). It sort of works if there are things that needs to be done after blocking_code() within main(); but blocking_code() is the last thing in main(), and it would cause the whole while loop to pause, until blocking_code() completes, before starting back from the top.

I don't know if this is possible, and if it is, how it's done; but the ideal scenario is this.

Run main(), then run blocking_code() in its own instance. As if executing another .py file. So once the loops reaches blocking_code() in main(), it triggers the blocking_code.py file, and whilst blocking_code.py script runs, the while loops continues from the top again.

If by the time on the 2nd pass in the while loop, it reaches blocking_code() again and the previous run has not complete; another instance of blocking_code() will run on its own instance, independently.

Does what I say make sense? Is it possible to achieve the desired outcome?

Thank you!

Solution

This is possible with threads. So you don't block your main loop, you'll need to wrap your thread in an asyncio task. You can wait for return values once your loop is finished if you need to. You can do this with a combination of asyncio.create_task and asyncio.to_thread

import aiohttp
import asyncio
import time

def blocking_code():
    print('Starting blocking code.')
    time.sleep(5)
    print('Finished blocking code.')

async def main():
    blocking_code_tasks = []

    while True:
        session_timeout = aiohttp.ClientTimeout()
        async with aiohttp.ClientSession() as session:
        
            print('Executing GET.')
            result = await session.get('https://www.example.com')
        
            blocking_code_task = asyncio.create_task(asyncio.to_thread(blocking_code))
            blocking_code_tasks.append(blocking_code_task)

    #do something with blocking_code_tasks, wait for them to finish, extract errors, etc.

asyncio.run(main())

The above code runes blocking code in a thread and then puts that into an asyncio task. We then add this to the blocking_code_tasks list to keep track of all the currently running tasks. Later on, you can get the values or errors out with something like asyncio.gather

Answered By - Matt Fowler