[FIXED] How can I catch an error in aiohttp get request?

Issue

Let's say I have some list of URLs to get some data from them. I try to do it in async way, but one of the URLs is incorrect. How can I catch this error and is it possible to change the URL address after catching the error to send the request again? I use the following code to get data using asyncio and aiohttp:

import asyncio
import aiohttp

urls = ["a", "b", "c"] # some list of urls
results = []

def get_tasks(session):
    tasks = []
    for url in urls:
        tasks.append(asyncio.create_task(session.get(url, ssl=False)))
    return tasks


async def get_symbols():
    async with aiohttp.ClientSession() as session:
        tasks = get_tasks(session)
        responses = await asyncio.gather(*tasks)
        for response in responses:
            results.append(await response.json())


asyncio.run(get_symbols())

And then I get the next error:

ContentTypeError: 0, message='Attempt to decode JSON with unexpected mimetype: ', url=URL('b')

How could I catch this error to continue the whole process and if it is possible to fix "b" to other correct URL (let's say "bb") and send request again?

Solution

The easiest way is to put try...except block around await response.json() and if it throws exception change the URL and schedule it again. For more complex task use for example asyncio.Queue.

import asyncio
import aiohttp

urls = [
    "https://reqbin.com/echo/get/json?1",
    "https://reqbin.com/echo/get/json?2",
    "https://reqbin.com/echo/get/json-BAD",
]
results = []


def get_tasks(session, urls):
    tasks = []
    for url in urls:
        tasks.append(asyncio.create_task(session.get(url, ssl=False)))
    return tasks


async def get_symbols():
    async with aiohttp.ClientSession() as session:
        while urls:
            for task in asyncio.as_completed(get_tasks(session, urls)):
                response = await task
                urls.remove(str(response.url))
                try:
                    data = await response.json()
                    print(response.url, data)
                    results.append(data)
                except Exception as e:
                    new_url = str(response.url).split("-")[0]
                    print(
                        f"Error with URL {response.url} Attempting new URL {new_url}"
                    )
                    urls.append(new_url)


asyncio.run(get_symbols())

Prints:

https://reqbin.com/echo/get/json?2 {'success': 'true'}
https://reqbin.com/echo/get/json?1 {'success': 'true'}
Error with URL https://reqbin.com/echo/get/json-BAD Attempting new URL https://reqbin.com/echo/get/json
https://reqbin.com/echo/get/json {'success': 'true'}

Answered By - Andrej Kesely