[FIXED] How to change scrapy closespider itemcount while parsing

Issue

i am new to scrapy. is it possible to change the CLOSESPIDER_ITEMCOUNT while the spider is running?

class TestSpider(scrapy.Spider):
    name = 'tester'
    custom_settings = {'CLOSESPIDER_ITEMCOUNT': 100,}

    def start_requests(self):
        urls = ['https://google.com', 'https://amazon.com']
        for url in urls:
            yield scrapy.Request(url, callback=self.parse)


    def parse(self, response):
        if response.xpath('//*[id="content"]') or True: # only for testing
            # set CLOSESPIDER_ITEMCOUNT to 300
        # rest of code

I want to be able to change the value on an "if condition" in the parse method

Solution

You can get access to the crawler settings object, unfreeze the settings, change the value and then freeze the settings object again. Please note that since this is not documented in the docs, it may have unexpected effects.

class TestSpider(scrapy.Spider):
    name = 'tester'
    custom_settings = {'CLOSESPIDER_ITEMCOUNT': 100,}

    def start_requests(self):
        urls = ['https://google.com', 'https://amazon.com']
        for url in urls:
            yield scrapy.Request(url, callback=self.parse)


    def parse(self, response):
        if response.xpath('//*[id="content"]') or True: # only for testing
            self.crawler.settings.frozen = False
            self.crawler.settings.set("CLOSESPIDER_ITEMCOUNT", 300)
            self.crawler.settings.frozen = True
        # add the rest of the code

Answered By - msenior_