[FIXED] How to use an asyncio loop inside another asyncio loop

Issue

I have been trying all kinds of things to be able to use an asyncio loop inside another asyncio loop. Most of the time my test just end in errors, such as:

RuntimeError: This event loop is already running

My example code below is just the base test I started with, so you can see the basics of what I am trying to do. I tried so many things after this test, it was just too confusing, so I figured I should keep it simple when asking for help. If anyone can point me in the right direction, that would be great. Thank you for your time!

import asyncio

async def fetch(data):
    message = 'Hey {}!'.format(data)
    other_data = ['image_a.com', 'image_b.com', 'image_c.com']
    images = sub_run(other_data)
    return {'message' : message, 'images' : images}

async def bound(sem, data):
    async with sem:
        r = await fetch(data)
        return r

async def build(dataset):
    tasks = []
    sem = asyncio.Semaphore(400)

    for data in dataset:
        task = asyncio.ensure_future(bound(sem, data))
        tasks.append(task)

    r = await asyncio.gather(*tasks)
    return r

def run(dataset):
    loop = asyncio.get_event_loop()
    future = asyncio.ensure_future(build(dataset))
    responses = loop.run_until_complete(future)
    loop.close()
    return responses

async def sub_fetch(data):
    image = 'https://{}'.format(data)
    return image

async def sub_bound(sem, data):
    async with sem:
        r = await sub_fetch(data)
        return r

async def sub_build(dataset):
    tasks = []
    sem = asyncio.Semaphore(400)

    for data in dataset:
        task = asyncio.ensure_future(sub_bound(sem, data))
        tasks.append(task)

    r = await asyncio.gather(*tasks)
    return r

def sub_run(dataset):
    loop = asyncio.get_event_loop()
    future = asyncio.ensure_future(sub_build(dataset))
    responses = loop.run_until_complete(future)
    loop.close()
    return responses

if __name__ == '__main__':
    dataset = ['Joe', 'Bob', 'Zoe', 'Howard']
    responses = run(dataset)
    print (responses)

Solution

Running loop.run_until_compete inside a running event loop would block the outer loop, thus defeating the purpose of using asyncio. Because of that, asyncio event loops aren't recursive, and one shouldn't need to run them recursively. (See this issue for details.) Instead of creating an inner event loop, await a task on the existing one.

In your case, remove sub_run and simply replace its usage:

images = sub_run(other_data)

with:

images = await sub_build(other_data)

And it will work just fine, running the sub-coroutines and not continuing with the outer coroutine until the inner one is complete, as you likely intended from the sync code.

Answered By - user4815162342