[FIXED] httpx: How to access specific responses from gathered request tasks?

Issue

I want to use HTTPX (within FastAPI, if that matters) to make asynchronous http requests to an outside API and store the responses as individual variables for processing in slightly different ways depending on which URL was fetched. I'm modifying the code from this StackOverflow answer.

import asyncio
import httpx

async def perform_request(client, url):
    response = await client.get(url)
    return response.text

async def gather_tasks(*urls):
    async with httpx.AsyncClient() as client:
        tasks = [perform_request(client, url) for url in urls]
        result = await asyncio.gather(*tasks)
        return result

async def f():
    url1 = "https://api.com/object=562"
    url2 = "https://api.com/object=383"
    url3 = "https://api.com/object=167"
    url4 = "https://api.com/object=884"
    result = await gather_tasks(url1, url2, url3, url4)
    # print(result[0])
    # print(result[1])
    # DO THINGS WITH url2, SOMETHING ELSE WITH url4, ETC.

if __name__ == '__main__':
    asyncio.run(f())

What's the best way to access the individual responses? (If I use result[n] I wouldn't know which response I'm working with.)

And I'm pretty new to httpx and async operations in general so please share if you have any suggestions for how to achieve it in a better way.

Solution

Regardless of AsyncIO, I would probably put the logic inside gather_tasks. There you know the response, and you can define all the if else logic you want to proceed with the right path.

In my opinion you have two options:

---

1 - Process the request right away

In this case f would only initialize the urls and trigger the processing, everything else would happen inside gather_tasks.

2 - "Enrich" the response

In gather_tasks you can understand which kind of operation to do next, and "attach" to the response some sort of code to define it. For example, you could return a dict with two keys: response and operation. This would be the most explicit way of doing this, but you could also use a list or a tuple, you just need to know where the response and the "next step code" is within them.

This is useful if the further processing must happen later instead of right away.

---

Makes sense?

Answered By - Sushi2all