[FIXED] Scrapy - User session timeout occurs and logouts user while scraping a website

Issue

Why does the user is being loggedout, while in the middle of scraping? I can not share my code, i just wanted to know why does this behaviour occurs -

• Is it because of cookies? ( I have not passed any cookies manually, as scrapy handles the cookies by default)

• Is it that certain headers are missing in requests?(I have not passed any headers explictly for requests)

• Or is it due to any other reasons?

• If it is because of cookies, how can i pass them for each requests?

• If it is because of headers, what are the most important headers that need to be passed for each requests?

Any suggestions will be appreciated.

Solution

If you log in, yes you need to handle the cookies, For more information about the cookies, please look at this answer.

It can be also because your session is expired on the site, so when you have this problem you need to re-log on the site.

For the headers, you need to inspect the network and see which headers you need for your requests.

Answered By - parik