Issue
Hi I am new to scrapy and I am trying to scrape categories and URLs of categories from alibaba's Product By categories page. I am trying to scrape it and put it on a CSV file.
Here is the view I want to give when I open it in a spreadsheet is:-
categories categories_urls
Agricultural Growing Media its URL
Animal Products its URL
. .
. .
. .
Code:-
# -*- coding: utf-8 -*-
import scrapy
class AlibabaCatagoriesSpider(scrapy.Spider):
name = 'alibaba_catagories'
allowed_domains = ['alibaba.com']
start_urls = ['https://www.alibaba.com/Products?spm=a2700.8293689.scGlobalHomeHeader.352.2ce265aa7GOmOF']
def parse(self, response):
a = response.css('ul.sub-item-cont')
for catag in a:
item = {
'categories': catag.css('li>a::text').extract(),
'categories_url': catag.css('li>a::attr(href)').extract()
}
yield item
Problems
- \n and white space are scraped while scraping categories.
- The data is not scraped in a desirable format
How can you help
- Modify the code so we can have the
- Give the trick to remove \n and white space while scraping
desirable format.
Solution
Pretty easy with Scrapy:
def parse(self, response):
for category_node in response.xpath('//ul[contains(@class, "sub-item-cont")]/li/a'):
item = {
'categories': category_node.xpath('./text()').extract_first().strip(),
'categories_url': category_node.xpath('./@href').extract_first()
}
yield item
Answered By - gangabass
0 comments:
Post a Comment
Note: Only a member of this blog may post a comment.