Issue
I am trying to scrape data from a list of urls. And for the urls which are not giving response status code 200 I do need some other data.
settings.py
HTTPERROR_ALLOWED_CODES = [404,403,301,302]
Code
def parse(self, response):
item = ScrapedataItem()
if response.status != 404 or response.status != 403 or response.status != 301 or response.status != 302:
item["url"] = response.url
item["status"] = response.status
item["html_data"] = response.text
else:
item["url"] = response.url
item["status"] = response.status
item["html_data"] = "Site Error"
Log
'downloader/response_status_count/200': 231,
'downloader/response_status_count/301': 12,
'downloader/response_status_count/302': 38,
'downloader/response_status_count/404': 4,
Here else
part is not executing. I want to make the else portion to execute if error codes come.
Solution
Answer to this is
No need to set in settings.py
We can handle error by adding list of error codes handle_httpstatus_list
to spider file
class CrawldataSpider(scrapy.Spider):
name = 'crawldata'
handle_httpstatus_list = [404,403,301,302]
def parse(self, response):
item = ScrapedataItem()
if response.status in (404,) or response.status in (403,) or response.status in (301,) or response.status in (302,):
item["url"] = response.url
item["status"] = response.status
item["html_data"] = "Site Error"
else:
item["url"] = response.url
item["status"] = response.status
item["html_data"] = response.text
Answered By - imhans4305
0 comments:
Post a Comment
Note: Only a member of this blog may post a comment.