Issue
i'm trying to scrape the website that contains loading screens. when i browse the website it shows loading.. for a sec and then it loads up. But the problem is when i try to scrape it using scrapy
it gives me nothing (probably because of that loading). can i solve the problem using scrapy
or should i use some other tools?
here's the link to the website if you wanna see https://www.graana.com/project/601/lotus-lake-towers
Solution
As it is sending a GET request to get information about the property , you should mimic the same in your code. (You can observe the GET call under console -> Network -> XHR )
# -*- coding: utf-8 -*-
import scrapy
class GranaSpider(scrapy.Spider):
name = 'grana'
allowed_domains = 'www.graana.com'
start_urls = ['https://www.graana.com/api/area/slug/601']
def parse(self, response):
# for url in allurlList:
scrapy.http.Request(response.url, method='GET' , dont_filter=False)
print(response.body)
#convert json response to array and save to your storage system
Output is in json format, convert it to your convenience.
Answered By - Deepa MG
0 comments:
Post a Comment
Note: Only a member of this blog may post a comment.