[FIXED] How do I call an async function with method run_in_executor in Python/Tornado?

Issue

I have a python file where I have a tornado request handler class with the following relevant code:

executor = concurrent.futures.ThreadPoolExecutor(max_workers = 20)
from tornado.platform.asyncio import AnyThreadEventLoopPolicy
asyncio.set_event_loop_policy(AnyThreadEventLoopPolicy())

class MainHandler(tornado.web.RequestHandler):
   async def connect(self):
      # some code
      loop = asyncio.get_event_loop()
      await loop.run_in_executor(executor, self.connect_function)

   async def connect_function(self):
      #some code with an await operation somewhere here

So my main goal is to be able to use the request handler with threads. The way I found I could do this is with the method run_in_executor. The problem here is that in the connect method I want to await the end of connect_function which also is async, this usually raises an error: RuntimeWarning: coroutine 'MainHandler.connect_function' was never awaited . My questions are if is this the only way I can make threads work in this tornado request handler, and if so, if there's a way I can wait the execution of an async function with the method run_in_executor.

Solution

You can't run async functions in a ThreadPoolExecutor (unless you also run another event loop there, which is not usually what you want). You could schedule a callback on the main event loop and then wait for it on the thread, but that's complex. As a general rule, when you're using an async framework like Tornado, you want to stay on the event loop thread as much as possible, and only send things out to a ThreadPoolExecutor when necessary (and then let those functions return to the main thread instead of trying to call back to async code from the thread).

Answered By - Ben Darnell