[FIXED] Can we use locks in Scrapy? Are they ever required as Scrapy is single threaded?

Issue

In my Item Pipeline to add data in database, during processing an item, connection to server can drop randomly and I need to re-establish connection. Now in the open_connection function, do I need to make sure that only one thread will create a new connection and all the concurrent request wont start creating a new connection ?

As scrapy is single threaded, I guess I would not need any locks in this scenario as one thread would re-establish connection and all the next incoming callback to process_item function would have an established connection. If this is wrong, can I use all usual Python locks with scrapy?

Solution

Scrapy is not single threaded but you still do not need locks.

Internally scrapy engine can launch multiple requests to the target server at once, but it maintains the order/sequence of the all the internal threads making it seemingly single threaded to the user.

So yes you will not need any locks because at any time only 1 thread would be allowed to process the pipeline.

Answered By - Ahsan Nawaz