Issue
This is my first Scrapy program, i couldn't see the results even executed without errors.
import scrapy
class sageSpyder(scrapy.Spider):
name= 'sagespyder'
start_url = [
'https://journals.sagepub.com/action/doSearch?AllField=cancer'
]
def parse(self, response):
for item in response.xpath("//div//ol[@class='search-results']//li"):
yield {
'art_title': item.xpath(".//article//h2//span//a//text()").extract_first()
}
Solution
You have a typo in start_urls:
start_urls = [
'https://journals.sagepub.com/action/doSearch?AllField=cancer'
]
Also you'll need (in settings.py
):
ROBOTSTXT_OBEY = False
Answered By - gangabass
0 comments:
Post a Comment
Note: Only a member of this blog may post a comment.