[FIXED] Discord.py bot, can I do a heavy task "off to the side" so I don't lag inputs?

Issue

I have a Discord bot in Python / Discord.py where people can enter commands, and normally the bot responds very quickly.

However the bot is also gathering/scraping webdata every iteration of the main loop. Normally the scraping is pretty short and sweet so nobody really notices, but from time to time the code is set up to do a more thorough scraping which takes a lot more time. But during these heavy scrapings, the bot is sort of unresponsive to user commands.

@bot.command()
async def sample_command(ctx):
    # may actually take a while for this command to respond if we happen to be
    # in the middle of a heavier site scrape
    await ctx.channel.send("Random message, something indicating bot has responded")  

async def main_loop():
    sem = asyncio.Semaphore(60)
    connector = aiohttp.TCPConnector(limit=60)

    async with aiohttp.ClientSession(connector=connector, headers=headers) as session:
        while True:
            # main guts of loop here ... 

            scrapers = [scraper_1(session, sem), scraper_2(session, sem), ...]
            data = list(chain(*await asyncio.gather(*scrapers)))  # this may take a while

            # do stuff with data

Is there a way to sort of have it go "Hey, you want to do a heavy scrape, fine go process it elsewhere - meanwhile let's continue with the main loop and I'll hook back up with you later when you're done and we'll process the data then", if that makes sense?

I mainly want to separate this scraping step so it's not holding up the ability for people to actually interact with the rest of the bot.

Solution

You can use asyncio.create_task() to spawn the scraping in the "background":

async def scrape_and_process(...):
    scrapers = [scraper_1(session, sem), scraper_2(session, sem), ...]
    data = list(chain(*await asyncio.gather(*scrapers)))  # this may take a while
    # do stuff with data

async def main_loop():
    sem = asyncio.Semaphore(60)
    connector = aiohttp.TCPConnector(limit=60)

    async with aiohttp.ClientSession(connector=connector, headers=headers) as session:
        while True:
            # main guts of loop here ... 

            # initiate scraping and processing in the background and
            # proceed with the loop
            asyncio.create_task(scrape_and_process(...))

Answered By - user4815162342