[FIXED] What is the proper way to start bunch of coroutines in loop using python asyncio.Semaphore?

Issue

I want to make a function, which will start bunch of coroutines, and wait for all of them to get done, by using asyncio Semaphore class. After processing this bunch of coroutines, I want to setup a timeout, to wait until running the next bunch.

I had tried to make this by using two for loops, but I'm not quite sure, that this is the best way to solve this problem. Here is the code:

async def start(self):
        sem = asyncio.Semaphore(10)
        latest_block_number = await self.w3.eth.block_number
        tasks = []
        for i in range(latest_block_number, latest_block_number-self.blocks_needed, -10):
            for j in range(i, i - 10, -1):
                task = asyncio.create_task(self.get_transactions_of_block(current_block_number=j, sem=sem))
                tasks.append(task)

            await asyncio.gather(*tasks)
            tasks.clear()
            await asyncio.sleep(60)

Can you please help me with your ideas? Thank you!

Solution

you can do this with asyncio.gather(). This will bunch everything together (or in technical terms, allows you to run multiple coroutine functions concurrently using asynchronous I/O) and ensure that they are run. Then you can invoke a synchronous sleep.

Also, there is a link to the docs: https://docs.python.org/3/library/asyncio-task.html

This is some code i did a while ago to test, but i have added a for loop that also waits 5 seconds between each loop.

import asyncio
import time


def main():
    ''' main entry point for the program '''
    # create the event loop and add to the loop
    # or run directly.

    asyncio.run(main_async())
    return

async def main_async():
    ''' the main async function '''
    # await foo()
    # await bar()
    for i in range(3):
        await asyncio.gather(foo(), bar())
        print('wait a bit..')
        time.sleep(5)
    return

async def foo():
    print('before the sleep')
    await asyncio.sleep(2)
    # time.sleep(2)
    print('world')
    return

async def bar():
    print('hello')
    await asyncio.sleep(0)
    return



if __name__=='__main__':
    ''' This is executed when run from the command line '''
    main()

The process above returns this:

hello
world
wait a bit..
before the sleep
hello
world
wait a bit..
before the sleep
hello
world
wait a bit..

Answered By - D.L