Issue
I am trying to use scrapy to scrape through a list of urls and get as an output only the urls that a specific keyword was found. I have tried the if statement through shell and it seems to be working --I tried if(..): exit() and it did exit the shell-- but the output of the following is an empty file (using the same url that returned the if statement as true in shell).
def parse(self, response):
filename = f'file.txt'
if (response.css('*').re('keyword')):
with open(filename, 'wb') as f:
f.write(response.url)
self.log(f'Saved file {filename}')
Solution
You open file in write-binary mode and try to write strings to it, this should raise an error.
Instead open file in append mode: with open(filename, 'a') as f:
Or just in write mode: with open(filename, 'a') as f:
Answered By - basckerwil
0 comments:
Post a Comment
Note: Only a member of this blog may post a comment.