[FIXED] Display an item search in ajax format from an ecommerce website

Issue

I am trying to scrape an ecommerce website (Lazada.sg) and I found a Github code based of on scrapy: https://github.com/talk2div/lazada-scraper. As I'm tinkering how he developed his code, however, I cannot replicate displaying the same URL search in ajax format (correct me if I'm wrong). Here is a sample of the URL for request in scrapy:https://www.lazada.sg/mother-baby/?ajax=true&page=1&spm=a2o42.searchlistcategory.cate_5b6ee3f0Npltyg.

The searches he made are for baby item searches. I am trying to replicate that for Lego items. I would be glad if I can have some help on displaying the URL in the same format as he did for scrapy so I can re-use his code for my own use case. Thanks

Solution

That is because the links he is querrying is part of an enumerated category listed on the page. You just want to get the search results of a specific keyword so the query will look a little different like this:

...
page=1
def start_requests(self):
        yield scrapy.Request(url=f'https://www.lazada.sg/catalog/?_keyori=ss&ajax=true&from=input&isFirstRequest=true&page={self.page}&q=lego&spm=a2o42.searchlistcategory.search.go.d1c332ab2wBQx9')

This is the link for the first page.

https://www.lazada.sg/catalog/?_keyori=ss&ajax=true&from=input&isFirstRequest=true&page=1&q=lego&spm=a2o42.searchlistcategory.search.go.d1c332ab2wBQx9

Answered By - Alexander