[FIXED] Process multiple element using asyncio instead of threads

Issue

I have a program that uses python threads like this: starting 5 threads and start processing a list of elements.

def process_element(element):
   print(element)

jobs = Queue()
    
def do_stuff(q):
    while not q.empty():
        value = q.get()
        process_element(element=value)
        q.task_done()
 
for i in line: # my list of element
    jobs.put(i)
    
for i in range(5):
    worker = threading.Thread(target=do_stuff, args=(jobs))
    worker.start()

jobs.join()

How can I use asyncio to do the same work.

Solution

A literal translation of your code into asyncio would look like this:

import asyncio, random

async def process_element(element):
    print('starting', element)
    await asyncio.sleep(random.random())  # simulate IO-bound processing
    print('done', element)

async def do_stuff(q):
    while not q.empty():
        value = await q.get()
        await process_element(element=value)
        q.task_done()

async def main():
    jobs = asyncio.Queue()

    for i in range(20):
        await jobs.put(i)

    for i in range(5):
        asyncio.create_task(do_stuff(jobs))

    await jobs.join()

asyncio.run(main())

Note, however, that:

• Asyncio handles IO-bound tasks such as communicating with HTTP servers or remote databases, chat servers, etc. It doesn't handle CPU-bound tasks, where a long-running operation will block the whole event loop. (In such cases multithreading or multiprocessing are more appropriate.) For this reason "converting" multithreaded code to asyncio often fails.

• while not q.empty(): ... process ... is an anti-pattern in both threading and asyncio because it doesn't allow waiting for something to arrive in the queue. If you know all your items in advance, you don't need a queue in the first place, you could use ordinary lists.

Answered By - user4815162342