[FIXED] How to send private data in spider to pipeline?

Issue

Say, each time I run scrapy like below

scrapy crawl [spidername] -a file='filename'

I want send the filename to pipeline to specify the item storage location. Each time the location may be different so it can't define in settings.py.

The file save in spider as private var

def __init__(self,file):
  self.filename=file

How can I send the parameter to pipeline?

Solution

There are a few ways to do this. One way would be to define the filename as a parameter in your spider's init() method, and then pass it to your pipeline as an argument when you call the process_item() method.

Another way would be to define the filename as a class variable in your spider, and then access it from your pipeline as an attribute of the spider.

Here is an example of how you could do it using a class variable:

class MySpider(Spider):
    filename = 'filename'

    def __init__(self, *args, **kwargs):
        super(MySpider, self).__init__(*args, **kwargs)

    def parse(self, response):
        # do stuff

    def process_item(self, item, spider):
        item['filename'] = spider.filename
        return item

Answered By - Mohamed Elgazar