[FIXED] Why async folder traversal is slower than sequential

Issue

I am benchmarking a folder traversal with aiofiles and normal os python lib. Here is my code: This is the async version of my function

import os
import time
from typing import List
import aiofiles.os as aio_os
import asyncio

subfolders1= set()
async def traverrse_local_path() -> List[str]:  

    async def traverse(local_path, count=None):
        
        try:
            current_dirs_list = [local_path + '/' + f for f in os.listdir(local_path)]
            current_subfolders = [f for f in current_dirs_list if os.path.isdir(f)]

            if not current_subfolders:
                if any([await aio_os.path.isfile(f) for f in current_dirs_list]):
                    subfolders1.add(local_path)
                    return
            for p in current_subfolders:
                    await traverse(p, count)
        except WindowsError:
            pass



    start = time.perf_counter()
    # tasks = [traverse(f.path) for f in current_path_folders]
    path_to_proj = 'C:/Program Files'
    list_folders = [path_to_proj + '/' + f for f in  os.listdir(path_to_proj) if os.path.isdir(path_to_proj + '/' + f)]
    await asyncio.gather(*(traverse(f,count) for count,f in enumerate(list_folders)))
    end = time.perf_counter()
    print(end-start)

loop = asyncio.get_event_loop()
try:
    loop.run_until_complete(traverrse_local_path())
finally:
    loop.close()
print(len(subfolders1))

where this is my normal version - sequential of my function

subfolders_2 = set()
def traverrse_local_path() -> List[str]:
    
    def traverse(local_path, count=None):
        try:
            current_dirs_list = [local_path + '/' + f for f in  os.listdir(local_path)]
            current_subfolders = [f for f in current_dirs_list if os.path.isdir(f)]
            if not current_subfolders:
                if any([os.path.isfile(f) for f in current_dirs_list]):
                    subfolders_2.add(local_path)
                    return
            for p in current_subfolders:
                    traverse(p, count)
        except WindowsError:
            pass

    start = time.perf_counter()
    path_to_proj = 'C:/Program Files'
    traverse(path_to_proj)
    end = time.perf_counter()
    print(end - start)

traverrse_local_path()
print(len(subfolders_2))

and that's the time comparison. enter image description here

Why is that, I thought with async one it should go faster?

Solution

Because async has much more overhead, and in your case there is no advantage.

oslistdir() gives you the entire content of the directory, in one go. Then you traverse each directory in it, and with two methods: with async you are expecting that reading some directories may be much slower (and so with CPU wait time) then an other directory inside the same parent directory. So in such wait-time you can get extra job done. The fact: it is a wrong assumption on most cases. From same parent, you should be able to access each directories at same speed. So now you have just overhead.

There may be some cases where your assumption is true: different mounting points inside the traversal, but usually there are not so many mounting points. But if you have a fast and a slow hard-disk, you may see some differences (if you get at beginning such differences). Or in case of network disks, but usually same network same file server, so all packets should take the same time (so async may help just if you have local and network disks).

So: async is good when you have many tasks which should wait to be completed, and the waiting is not caused by access to the same resource. On disks you may get the first, but cache is efficient, and disks are fast, so short wait time which may be used for OS multitasking (so other processes), so not much advantage. But directory traversal very often fails second requirement. So you just have a lot of overhead.

Answered By - Giacomo Catenazzi