[FIXED] How to parse embedded links through Python Scrapy spider

Issue

I am trying to use python's scrappy to extract course catalog information from a website. The thing is, each course has a link to its full page and I need to iterate through those pages one by one to extract their information, which later, are fed to an SQL database. Anyhow, I don't know how to change the url's in the spider successively. here attached below is my code so far.

import scrapy

def find_between(s, first, last):
    try:
        start = s.index(first) + len(first)
        end = s.index(last, start)
        return s[start:end]
    except ValueError:
        return ""

class QuoteSpider(scrapy.Spider):
    name = 'courses'
    start_urls = [
        'http://catalog.aucegypt.edu/content.php?catoid=36&navoid=1738',

    ]

    def parse(self, response):
        # pages in span+ a
        all_courses = response.css('.width a')
        for course in all_courses:
            courseURL = course.xpath('@href').extract()
            cleanCourseURL = find_between(str(courseURL), "['", "']")
            fullURL = "http://catalog.aucegypt.edu/" + cleanCourseURL

            #iterate through urls
            QuoteSpider.start_urls += fullURL
            courseName = response.css('.block_content')


            yield {
                'courseNum': fullURL,
                'test': courseName
            }

Solution

Usually you need to yield this new URL and process it with corresponding callback:

def parse(self, response):
    # pages in span+ a
    all_courses = response.css('.width a')
    for course in all_courses:
        courseURL = course.xpath('@href').extract()
        cleanCourseURL = find_between(str(courseURL), "['", "']")
        fullURL = "http://catalog.aucegypt.edu/" + cleanCourseURL
        courseName = response.css('.block_content')
        yield scrapy.Request(
            url=fullURL,
            callback=self.parse_course,
            cb_kwargs={
                'course_name': courseName,
            },
        )

def parse_course(self, response, course_name):
    # parse you course here...

Answered By - gangabass