[FIXED] Using scrapy 2.7 ModuleNotFoundError: No module named 'scrapy.squeue'

Issue

I run my scrapy as a standalone script like this

if __name__ == "__main__":
    from scrapy.crawler import CrawlerProcess
    from scrapy.utils.project import get_project_settings

    s = get_project_settings()
    process = CrawlerProcess(s)
    process.crawl(MySpider)
    process.start()

my scraper was consuming huge memory so i thought of using these two custom settings.

SCHEDULER_DISK_QUEUE = "scrapy.squeue.PickleFifoDiskQueue"
SCHEDULER_MEMORY_QUEUE = "scrapy.squeue.FifoMemoryQueue"

But after adding these two custom settings and when i run my standalone spider I get error saying.

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/twisted/internet/defer.py", line 1696, in _inlineCallbacks
    result = context.run(gen.send, result)
  File "/usr/local/lib/python3.9/dist-packages/scrapy/crawler.py", line 118, in crawl
    yield self.engine.open_spider(self.spider, start_requests)
ModuleNotFoundError: No module named 'scrapy.squeue'

Any ideas what's the issue with this ?

Solution

ModuleNotFoundError: No module named 'scrapy.squeue'

You have a typo:

SCHEDULER_DISK_QUEUE = "scrapy.squeues.PickleFifoDiskQueue"
SCHEDULER_MEMORY_QUEUE = "scrapy.squeues.FifoMemoryQueue"

Answered By - SuperUser