[FIXED] Problems running web scraper in Spyder IDE

Issue

I have a code that is using Scrpay framework and here's the code

import scrapy
from scrapy.crawler import CrawlerProcess

class DemoSpider(scrapy.Spider):
    name = "DemoSpider"
    
    def start_requests(self):
        urls = ['http://quotes.toscrape.com/page/1/', 'http://quotes.toscrape.com/page/2/']
        for url in urls:
            yield scrapy.Request(url=url, callback=self.parse)
    
    def parse(self, response):
        page = response.url.split('/')[-2]
        filename = 'quotes-%s.html' % page
        with open(filename, 'wb') as f:
            f.write(response.body)
            self.log('Saved File %s' % filename)

process = CrawlerProcess()
process.crawl(DemoSpider)
process.start()

The code is working well when running like that from terminal (Windows 10 PowerShell) python demo.py. But I need to run the code using Spyder IDE. When trying I got an error like that

    ReactorBase.startRunning(self)

  File "C:\ProgramData\Anaconda3\lib\site-packages\twisted\internet\base.py", line 765, in startRunning
    raise error.ReactorNotRestartable()

ReactorNotRestartable

Solution

(Spyder maintainer here) Please go to the menu Run > Configuration per file and activate the option Execute in an external system terminal.

That will run your code in a regular Python interpreter, which will avoid the problems you're having to start the server that runs the scraper in our IPython console.

Answered By - Carlos Cordoba