[FIXED] Xpath returns null

Issue

I need to scrape the price of this page: https://www.asos.com/monki/monki-lisa-cropped-vest-top-with-ruched-side-in-black/prd/23590636?colourwayid=60495910&cid=2623

However it is always returning null:

My code:

'price' :response.xpath('//*[contains(@class, "current-price")]').get()

Can someone help please?

Thanks!

When Extracted using XHR:

How to retrieve price?

Solution

Your problem is not the xpath, it's that the price is being retrieved with XHR.

If you use scrapy sheel and type view(response) you can see that the price is not being generated:

Look at the source of the original webpage and search for the price:

Then use this url the scrape the price:

    def parse(self, response):
        import re
        price_url = 'https://www.asos.com' + re.search(r'window.asos.pdp.config.stockPriceApiUrl = \'(.+)\'', response.text).group(1)
        yield scrapy.Request(url=price_url,
                             method='GET',
                             callback=self.parse_price,
                             headers=self.headers)

    def parse_price(self, response):
        import json
        jsonresponse = json.loads(response.text)
        ...............
        ...............
        ...............

I couldn't get around 403 error with the headers I provided, but maybe you'll have more luck.

Edit:

In order to get the price from the json file there's actually no need for json.loads

    def parse_price(self, response):
        jsonresponse = response.json()[0]
        price = jsonresponse['productPrice']['current']['text']
        # You can also use jsonresponse.get() if you prefer
        print(price)

Output:

£10.00

Answered By - SuperUser