[FIXED] Asyncify string joining in Python

Issue

I have the following code snippet which I want to transform into asynchronous code (data tends to be a large Iterable):

transformed_data = (do_some_transformation(d) for d in data)
stacked_jsons = "\n\n".join(json.dumps(t, separators=(",", ":")) for t in transformed_data)

I managed to rewrite the do_some_transformation-function to be async so I can do the following:

transformed_data = (await do_some_transformation(d) for d in data)
async_generator = (json.dumps(event, separators=(",", ":")) async for t in transformed_data)
stacked_jsons = ???

What's the best way to incrementally join the jsons produced by the async generator so that the joining process is also asynchronous? This snippet is part of a larger I/O-bound-application which and has many asynchronous components and thus would profit from asynchifying everything.

Solution

The point of str.join is to transform an entire list at once.¹ If items arrive incrementally, it can be advantageous to accumulate them one by one.

async def join(by: str, _items: 'AsyncIterable[str]') -> str:
    """Asynchronously joins items with some string"""
    result = ""
    async for item in _items:
        if result and by:  # only add the separator between items
            result += by
        result += item
    return result

The async for loop is sufficient to let the async iterable suspend between items so that other tasks may run. The primary advantage of this approach is that even for very many items, this never stalls the event loop for longer than adding the next item.

This utility can directly digest the async generator:

stacked_jsons = join("\n\n", (json.dumps(event, separators=(",", ":")) async for t in transformed_data))

---

When it is know that the data is small enough that str.join runs in adequate time, one can directly convert the data to a list instead and use str.join:

stacked_jsons = "\n\n".join([json.dumps(event, separators=(",", ":")) async for t in transformed_data])

The [... async for ...] construct is an asynchronous list comprehension. This internally works asynchronously to iterate, but produces a regular list once all items are fetched – only this resulting list is passed to str.join and can be processed synchronously.

---

¹ Even when joining an iterable, str.join will internally turn it into a list first.

Answered By - MisterMiyagi