[FIXED] Continue the batch script commands when using scrapy crawl command in a batch file

Issue

I am using scrapy for crawling the information from a webpage. I have written the spider code which works fine. I also have some other python code files which refine the crawled data.

To put it all together, I have a batch file in which I first I use the "Scrapy Crawl" command, and after that I run my python file to refine the crawled information.

The problem is that the batch script stops at the end of the "Scrapy Crawl" command, and does not proceed with executing the lines after that in my batch file. How can I solve this problem?

Contents of the batch file:

CD "spiders_folder"
scrapy crawl mySpider -o outputData.json -t json
python refineData.py 

Solution

The batch file logic handles calls to other programs as subroutines and returns from there back to the originating batch file after their execution is finished. AFAICR the exception from the rule was if one batch file calls another batch file. In this case the execution is not handled as a subroutine but is given over to the other batch file.

Unless you use call to call the second batch file.

(I might be mistaken, this is all very long ago for me, so please don't hit me too hard if I'm wrong.)

If my guess is correct and this is the reason for your trouble, it is because scrapy is a batch file itself. Then just call it like this:

call scrapy crawl mySpider -o outputData.json -t json

Answered By - Alfe