[FIXED] getting 200 response from requests lib but not through Scrapy in python

Issue

I have tried to scrap data using scrapy spider in python to the targeted URL: https://www.accenture.com/ro-en/services/data-analytics-index#block-what-we-think but it returns the Error: twisted.python.failure.Failure builtins.ValueError: not enough values to unpack (expected 2, got 1)

But if i try to scrape data using the python requests library it works fine.

Solution

The above answer does not completely work but it gives me the idea. You can check the twisted lib class (https://github.com/twisted/twisted/blob/trunk/src/twisted/protocols/basic.py#L406-L422) which is defined as MAX_LENGTH = 16384

we just need to override this in our init.py file of scrapy spider project like Below,

from twisted.protocols.basic import LineReceiver

LineReceiver.MAX_LENGTH = 65536

Answered By - Manoj Bhatt