Issue
I have the following asyncio code:
async def fetch(url, session):
async with session.get(url, ssl=False) as response:
status = response.status
data = await response.json(content_type='text/html')
data_length = len(data)
return {"url": url, "status": status, "data_length": data_length, "data": data}
async def fetch_all(urls):
async with aiohttp.ClientSession(headers=vtheaders) as session:
tasks = [fetch(url, session) for url in urls]
results = await asyncio.gather(*tasks)
return results
Which get a list of URLs, and is running them. specifically, I'm using virustotal API.
Now, I get the following error:
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
And I was able to track the response from the API request which was error 500 but it wasn't received in a JSON so I guess that's why I get json is empty so decoder is failing...
This is the error:
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN"> <title>500 Internal Server Error</title> <h1>Internal Server Error</h1> <p>The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application.</p>
I don't know how I can avoid this bad URL inside my asyncio loop :( Any guidance?
Solution
You can check if status is 200
async def fetch(url, session):
async with session.get(url, ssl=False) as response:
status = response.status
if status == 200:
data = await response.json(content_type='text/html')
else:
data = '{}'
data_length = len(data)
return {"url": url, "status": status, "data_length": data_length, "data": data}
Answered By - Guy
0 comments:
Post a Comment
Note: Only a member of this blog may post a comment.