Issue
I have been trying to get this crawler working but I keep getting errors.
Can anyone suggest any ways to get it to run?
The main spider code is
import scrapy
from scrapy.spiders import Spider
from scrapy.selector import Selector
class gameSpider(scrapy.Spider):
name = "game_spider.py"
allowed_domains = ["*"]
start_urls = [
"http://www.game.co.uk/en/grand-theft-auto-v-with-gta-online-3-500-000-1085837?categoryIdentifier=706209&catGroupId="
]
def parse(self, response):
sel = Selector(response)
sites = sel.xpath('//ul[@class="directory-url"]/li')
items = []
for site in sites:
item = Website()
item['name'] = site.xpath('//*[@id="details301149"]/div/div/h2/text()').extract()
"""item['link'] = site.xpath('//a/@href').extract()
item['description'] = site.xpath('//*[@id="overview"]/div[3]()').re('-\s[^\n]*\\r')"""
items.append(item)
print items
return items
The item code is
import scrapy
class GameItem(Item):
name = Field()
pass
Solution
Your start_urls
link returns erorr 500.
There's no items.
In [7]: sites = response.xpath('//ul[@class="directory-url"]/li')
In [8]: sites
Out[8]: []
Answered By - Daniil Mashkin
0 comments:
Post a Comment
Note: Only a member of this blog may post a comment.