[FIXED] how can i run SCRAPY outside scrapy project folder

Issue

(1) i want to use process.crawl('spidername') to run a scrapy BUT wdir isn't the scrapy project directory
just like this:

workspace  
├─ scrapy_project  
│    ├─ project_name  
│        ├─ spiders
│        │     └─ myspider.py    #import errors(2)
│        ├─ items.py
│        └─ settings.py
└─ run.py   # (1)i want to run this file to run the spider

(2)another problem is that it has errors when i import scrapy_project.project_name.items in myspider.py
how can i import items.py into myspider.py

Solution

The most idiomatic way to do this is is to install the scrapy_project package into whatever environment you want to run it in, and then import your method that way. This allows you to use the scrapy code you wrote wherever you want to. For more information on packaging your code see https://packaging.python.org/en/latest/tutorials/packaging-projects/

Once you have your package scrapy you can just install it using a shell with:

pip3 install /path/to/scrapy

Answered By - Phoenix