Issue
Fairly new to Python, following indications from DataCamp and youtube tutorials I'm trying to run a spider to crawl a website and extract metadata from the most recent (thousands) videos.
So far my Spider looks like this:
class NaughtySpider(scrapy.Spider):
name = "naughtyspider"
allowed_domains = ["example.com"]
start_url = ("https://www.example.com/video?o=cm")
# start_requests method
def start_requests(self):
yield scrapy.Request(url = start_url,
callback = self.parse_video)
# First parsing method
def parse_video(self, response):
self.log('F i n i s h e d s c r a p i n g ' + response.url)
video_links = response.css('ul#videoCategory').css('li.videoBox').css('div.thumbnail-info-wrapper').css('span.title > a').css('::attr(href)') #Correct path, chooses 32 videos from page ignoring the links coming from ads
links_to_follow = video_links.extract()
for url in links_to_follow:
yield response.follow(url = url,
callback = self.parse_metadata)
#Continue through pagination
next_page_url = response.css('li.page_next > a.orangeButton::attr(href)').extract_first()
if next_page_url:
next_page_url = response.urljoin(next_page_url)
yield scrapy.Request(url=next_page_url, callback=self.parse_video)
# Second parsing method
def parse_metadata(self, response):
# Create a SelectorList of the course titles text
video_title = response.css('div.title-container > h1.title > span.inlineFree::text')
# Extract the text and strip it clean
video_title_ext = video_title.extract_first().strip()
# Extract views
video_views = response.css('span.count::text').extract_first()
# Extract tags
video_tags = response.css('div.tagsWrapper a::text').extract()
del video_tags[-1] #Eliminate '+' tag, which is for suggestions
# Extract Categories
video_categories = response.css('div.categoriesWrapper a::text').extract()
del video_categories[-1] #Same as tags
# Fill in the dictionary
yield {
'title': video_title_ext,
'views': video_views,
'tags': video_tags,
'categories': video_categories,
}
I follow the documentation where it introduces this seemingly simple way to export the scraped data
scrapy crawl quotes -o quotes.json
but when I run the equivalent code
scrapy crawl naughtyspider -o data.csv
I get the following error log:
2019-08-17 22:24:54 [scrapy.core.engine] ERROR: Error while obtaining start requests
Traceback (most recent call last):
File "C:\Users\bla\Anaconda3\lib\site-packages\scrapy\core\engine.py", line 127, in _next_request
request = next(slot.start_requests)
File "C:\Users\bla\naughty\naughty\spiders\NaughtySpider.py", line 11, in start_requests
yield scrapy.Request(url = start_url,
NameError: name 'start_url' is not defined
2019-08-17 22:24:54 [scrapy.core.engine] INFO: Closing spider (finished)
Specially frustrating considering it is being defined in the previous line of code. I've seen similar situations in other questions, but none seems to fit exactly the code I'm using.
Thanks in advance, and sorry if there are major mistakes affecting the code, the resources around do not seem to be beginner friendly at all (no specification of which terminal / shell they are using, for example, mostly using Mac, etc.).
Solution
You need to use self.
if you're referencing a class variable:
def start_requests(self):
yield scrapy.Request(url = self.start_url,
callback = self.parse_video)
Answered By - gangabass
0 comments:
Post a Comment
Note: Only a member of this blog may post a comment.