[FIXED] Run a scrapy command inside py file with same venv

Issue

I want to run scrapy in a flask webservice (deployment mode) but when using os.system it doesn't run scrapy in the same virtualenv that I run the webservice. I don't have the same problem when I run it on local host.

subprocess package has the same issue and I don't have access to venv path.

Is there any way to do it?

os.chdir(SCRAPYFILE_PATH)
os.system(f"scrapy crawl spider_name -o file.json")

Solution

It worked using os and sys:

import sys 
env_path = os.path.dirname(sys.executable)
...
os.chdir(SCRAPYFILE_PATH)
os.system(f"{env_path}/scrapy crawl spider_name -o file.json")

Answered By - Violet