[FIXED] Scrapy xpath not able to extract

Issue

I am trying to extract data from link, I used

scrapy shell "https://www.newegg.com/Product/Product.aspx?Item=06T-0045-00045"

I got correct response, but can't get the Xpath to work,i.e getting the price with response.xpath('//li[@class="price-current"]') returns empty, I tried also response.xpath('//*[@id="landingpage-price"]/div/div/ul') but also empty, When I use response.xpath('//*[@id="landingpage-price"]') it works but anything deeper returns empty.

Solution

You always need to check source HTML (Ctrl+U). There is <meta itemprop='price' content='78.23' /> in the source. So simple:

response.xpath('//meta[@itemprop="price"]/@content').extract_first()

will work.

Answered By - gangabass