[FIXED] Scrapy - Duplicates filtering and download delays

Issue

I was wondering, how Scrapy deals with duplicates when a DOWNLOAD_DELAY is specified in the settings. For example, if I add a DOWNLOAD_DELAY of 5 seconds, Scrapy will wait 5 seconds between each request. However if Scrapy faces a duplicate request which should be ignored, does Scrapy will wait 5 seconds before the next request ?

Solution

Yes scrapy wait 5 second for next request. and if you're set duplicate_filter=True then it waiit for 5 second after call next request. no matter if req are duplicate or not scrapy always wait for next 5 second if you're duplicate filter.

Answered By - Viren Ramani