Issue
I'm extremely new to python and scrapy. I've tried running existing code and I'm getting these errors thrown. I'm running the latest version of scrapy on windows 10 and using Visual Code Studio to run my tests in, etc.
Terminal Debug
2020-12-07 20:52:52 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.newegg.com/robots.txt> (referer: None)
2020-12-07 20:52:52 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.newegg.com/asus-radeon-rx-6800-rx6800-16g/p/N82E16814126473> (referer: None)
2020-12-07 20:52:52 [scrapy.core.scraper] ERROR: Spider error processing <GET https://www.newegg.com/white-bluedio-ht-wireless/p/0TH-0357-00002?Item=9SIACD56WH3830> (referer: None)
Traceback (most recent call last):
File "C:\ProgramData\Miniconda3\lib\site-packages\twisted\internet\defer.py", line 654, in _runCallbacks
current.result = callback(current.result, *args, **kw)
File "C:\ProgramData\Miniconda3\lib\site-packages\scrapy\spiders\__init__.py", line 90, in _parse
return self.parse(response, **kwargs)
File "C:\ProgramData\Miniconda3\lib\site-packages\scrapy\spiders\__init__.py", line 93, in parse
raise NotImplementedError(f'{self.__class__.__name__}.parse callback is not defined')
NotImplementedError: NeweggSpider.parse callback is not defined
egg_spider.py
class NeweggSpider(scrapy.Spider):
name = "newegg"
USER_AGENT = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) " \
"Chrome/43.0.2357.130 Safari/537.36 "
allowed_domains = ['newegg.com']
start_urls = [
"https://www.newegg.com/white-bluedio-ht-wireless/p/0TH-0357-00002?Item=9SIACD56WH3830", ]
def parse(self, response):
# Finding Product Status.
my_list = []
try:
product = response.xpath("//*[@class='btn btn-primary btn-wide']//text()").get().strip()
my_list = [product]
print(f"Product is Currently: {my_list}")
except AttributeError:
pass
Any help in the most layman's terms possible. Thanks!
Solution
You need to indent your code as per gangabass comment.
import scrapy
class NeweggSpider(scrapy.Spider):
name = "newegg"
USER_AGENT = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) " \
"Chrome/43.0.2357.130 Safari/537.36 "
allowed_domains = ['newegg.com']
start_urls = [
"https://www.newegg.com/white-bluedio-ht-wireless/p/0TH-0357-00002?Item=9SIACD56WH3830", ]
def parse(self, response):
# Finding Product Status.
my_list = []
try:
product = response.xpath("//*[@class='btn btn-primary btn-wide']//text()").get().strip()
my_list = [product]
print(f"Product is Currently: {my_list}")
except AttributeError:
pass
Answered By - Samsul Islam
0 comments:
Post a Comment
Note: Only a member of this blog may post a comment.