Issue
I need to change the image link in case if current image URLS return 404 code I have implemented own pipeline by extending FilesPipeline.
I have supposed the method media_failed
will be called it we got 404 code, but it didn't happen.
in the method item_completed
I see that results
for failed URL contains the following info
<class 'tuple'>: (False, <twisted.python.failure.Failure scrapy.pipelines.files.FileException: download-error>)
in this case I have to update origin image link and retry downloading
I see the following info in logs:
[scrapy.pipelines.files] WARNING: File (code: 404): Error downloading file from <GET https://any_dummy_link.jpg> referred in <None>
Solution
Request the image URL. If response.status is 404 you can handle it differently.
Answered By - Keeno
0 comments:
Post a Comment
Note: Only a member of this blog may post a comment.