[FIXED] How to check response status for http error codes using Scrapy?

Issue

I want to check the response status and export it to CSV file using Scrapy. I tried with response.status but it only shows '200' and exports to the CSV file. How to get other status codes like "404", "502" etc.

def parse(self, response):
        yield {
            'URL': response.url,
            'Status': response.status
        }

Solution

In your settings you can adjust these to make sure certain error codes are not automatically filtered by scrapy.

HTTPERROR_ALLOWED_CODES

Default: []

Pass all responses with non-200 status codes contained in this list.

HTTPERROR_ALLOW_ALL

Default: False

Pass all responses, regardless of its status code.

settings.py

HTTPERROR_ALLOW_ALL = True

HTTPERROR_ALLOWED_CODES = [500, 501, 404 ...]

Answered By - Alexander