[FIXED] How to run same function twice with different request.meta in Scrapy

Issue

I have one function which needs to be run twice with a different request.meta in scrapy

request = scrapy.Request(tournament_url, callback=self.parse_tournament)
request.meta['data'] = team1_data
yield request

request1 = scrapy.Request(tournament_url, callback=self.parse_tournament)
request1.meta['data'] = team2_data
yield request1

As of now, only the first request is working!

Solution

You will want to include dont_filter in your 2nd Request to avoid the Scrapy DupeFilter dropping the already-seen URL:

request1 = scrapy.Request(tournament_url, callback=self.parse_tournament,
                          dont_filter=True)
request1.meta['data'] = team2_data
yield request

Answered By - mdaniel