[FIXED] Unable to understand the ValueError: invalid literal for int() with base 10: 'تومان'

Issue

My crawler isn't working properly and I can't find what is the solution to it.

Here is the related part of my spider:

def parse(self, response):
        original_price=0
        discounted_price=0
        star=0
        discounted_percent=0
        try:
            for product in response.xpath("//ul[@class='c-listing__items js-plp-products-list']/li"):
                title= product.xpath(".//div/div[2]/div/div/a/text()").get()
                if product.xpath(".//div/div[2]/div[2]/div[1]/text()"):
                    star= float(str(product.xpath(".//div/div[2]/div[2]/div[1]/text()").get()))
                if product.xpath(".//div/div[2]/div[3]/div/div/div[1]/span/text()"):
                    discounted_percent = int(str(product.xpath(".//div/div[2]/div[3]/div/div/div[1]/span/text()").get().strip()).replace('٪', ''))
                if product.xpath(".//div/div[2]/div[3]/div/div/div/text()"):
                    discounted_price= int(str(product.xpath(".//div/div[2]/div[3]/div/div/div/text()").get().strip()).replace(',', ''))
                if product.xpath(".//div/div[2]/div[3]/div/div/del/text()"):
                    original_price= int(str(product.xpath(".//div/div[2]/div[3]/div/div/del/text()").get().strip()).replace(',', ''))
                    discounted_amount= original_price-discounted_price
                else:
                    original_price= print("not available")
                    discounted_amount= print("not available")
                url= response.urljoin(product.xpath(".//div/div[2]/div/div/a/@href").get())

This is my log:

2020-10-21 16:49:56 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (301) to <GET https://www.digikala.com/search/category-book/> from <GET https://www.digikala.com/search/category-book>
2020-10-21 16:49:57 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.digikala.com/search/category-book/> (referer: None)
2020-10-21 16:49:57 [scrapy.core.scraper] ERROR: Spider error processing <GET https://www.digikala.com/search/category-book/> (referer: None)
Traceback (most recent call last):
  File "C:\Users\shima\anaconda3\envs\virtual_workspace\lib\site-packages\scrapy\utils\defer.py", line 102, in iter_errback
    yield next(it)
  File "C:\Users\shima\anaconda3\envs\virtual_workspace\lib\site-packages\scrapy\spidermiddlewares\offsite.py", line 29, in process_spider_output
    for x in result:
  File "C:\Users\shima\anaconda3\envs\virtual_workspace\lib\site-packages\scrapy\spidermiddlewares\referer.py", line 339, in <genexpr>
    return (_set_referer(r) for r in result or ())
  File "C:\Users\shima\anaconda3\envs\virtual_workspace\lib\site-packages\scrapy\spidermiddlewares\urllength.py", line 37, in <genexpr>
    return (r for r in result or () if _filter(r))
  File "C:\Users\shima\anaconda3\envs\virtual_workspace\lib\site-packages\scrapy\spidermiddlewares\depth.py", line 58, in <genexpr>
    return (r for r in result or () if _filter(r))
  File "C:\Users\shima\projects\digi_allbooks\digi_allbooks\spiders\allbooks.py", line 31, in parse
    discounted_percent = int(str(product.xpath(".//div/div[2]/div[3]/div/div/div[1]/span/text()").get().strip()).replace('٪', ''))
ValueError: invalid literal for int() with base 10: 'تومان'
2020-10-21 16:49:57 [scrapy.core.engine] INFO: Closing spider (finished)
2020-10-21 16:49:57 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 939,
 'downloader/request_count': 3,
 'downloader/request_method_count/GET': 3,
 'downloader/response_bytes': 90506,
 'downloader/response_count': 3,
 'downloader/response_status_count/200': 2,
 'downloader/response_status_count/301': 1,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2020, 10, 21, 13, 19, 57, 630044),
 'log_count/DEBUG': 3,
 'log_count/ERROR': 1,
 'log_count/INFO': 9,
 'log_count/WARNING': 1,
 'response_received_count': 2,
 'robotstxt/request_count': 1,
 'robotstxt/response_count': 1,
 'robotstxt/response_status_count/200': 1,
 'scheduler/dequeued': 2,
 'scheduler/dequeued/memory': 2,
 'scheduler/enqueued': 2,
 'scheduler/enqueued/memory': 2,
 'spider_exceptions/ValueError': 1,
 'start_time': datetime.datetime(2020, 10, 21, 13, 19, 55, 914304)}
2020-10-21 16:49:57 [scrapy.core.engine] INFO: Spider closed (finished)

I guess it says there is a string in an int() function which returns the ValueError but the XPath I'm using targets a number, not a string. I can't get the error correctly, so I don't find the solution. Can someone help me out, please?

Solution

In at least one of the iterations this line is scraping تومان instead of an integer

discounted_percent = int(str(product.xpath(".//div/div[2]/div[3]/div/div/div[1]/span/text()").get().strip()).replace('٪', ''))

From a google search it seems this is a monetary unit. You need to work on your XPaths, or have the spider ignore this return as there isn't a discount in this item.

It seems this XPath may be a better option for your intention: (I haven't checked all items though)

product.xpath(".//div[@class="c-price__discount-oval"]/span/text()").get()

Answered By - renatodvc