Issue
from scrapy import Spider
from selenium import webdriver
from scrapy.selector import Selector
class FlipkartSpider(Spider):
name = 'flipkarttrial1'
allowed_domains = ['flipkart.com']
def start_requests(self):
self.driver = webdriver.Chrome('C:\\Users\\xyz\\chromedriver')
self.driver.get('https://www.flipkart.com/womens-footwear/heels/pr?sid=osp,iko,6q1&otracker=nmenu_sub_Women_0_Heels')
sel = Selector(text=self.driver.page_source)
prices = sel.xpath('//div/div[@class="_1vC4OE"]/text()').extract()
for price in prices:
print(price)
def parse(self, response):
pass
Here the scraper prints the price but when I use yield, it throws an error. I want to save prices to csv file. How can I save the data using 'yield'?
Solution
You have to yield
a proper Python dict like below
I guess you cannot yield items from strat_requests so just do a dummy request and then yield items in parse method
def start_requests(self):
yield scrapy.Request(url=“http://ip-api.com/json”, callback=self.parse)
def parse(self, response):
self.driver = webdriver.Chrome('C:\\Users\\xyz\\chromedriver')
self.driver.get('https://www.flipkart.com/womens-footwear/heels/pr?sid=osp,iko,6q1&otracker=nmenu_sub_Women_0_Heels')
sel = Selector(text=self.driver.page_source)
prices = sel.xpath('//div/div[@class="_1vC4OE"]/text()').extract()
for price in prices:
yield {“price": price}
Answered By - Umair Ayub
0 comments:
Post a Comment
Note: Only a member of this blog may post a comment.