Issue
I am trying store the html code into variable called response using cmdline.execute as shown in below code ,but it is unavailable to store and program code breaks at scrapy shell, can anyone tell me how to store raw html to variable
import scrapy
from scrapy import cmdline
linkedinnurl = "https://stackoverflow.com/users/5597065/adnan-stab=profile"
response = cmdline.execute("scrapy shell https://stackoverflow.com/users/5597065/adnan-s?tab=profile".split()))
print(response)
Solution
You can do like this to store raw html to variable:
class MySpider(scrapy.Spider):
def parse(self, res):
with open(dynamic_file_name_function(res.url), 'w') as f:
f.write(res.body)
if you don't need dynamic file name then just do :
class MySpider(scrapy.Spider):
def parse(self, res):
with open(your_file_path, 'w') as f:
f.write(res.body)
Answered By - Amrit
0 comments:
Post a Comment
Note: Only a member of this blog may post a comment.