Issue
Is there a way in scrapy how to repeat a request, if the response status code is 440?
Sometimes the server i am trying to scrape responds with 440 code and the I can´t scrape the data.
Thanks a lot!
Solution
Based on the scrapy documentation you can add and configure the RetryMiddleware
to behave the way you want.
The settings you will want to change are: RETRY_TIMES
and RETRY_HTTP_CODES
. Specially the second one where the defaults are: [500, 502, 503, 504, 408]
, so there you just need to add 440
to the list.
Check more details about the middleware here
Answered By - dethos
0 comments:
Post a Comment
Note: Only a member of this blog may post a comment.