Issue
I have this code here inside middlewares.py > process_request
:
def process_request(self, request, spider):
refreerr_list = [
'https://google.com',
'https://amazon.com',
'https://facebook.com',.....
]
referr = random.choice(refreerr_list)
request.header['referer'] = referr
As it should show referer but in the log, it shows:
2022-07-17 07:17:13 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.com/AMD-Ryze.....RO-3995WX/dp/B08V5HPXVY> (referer: None)
Do I have to add something in settings.py?
But it works when doing in settings.py
:
DEFAULT_REQUEST_HEADERS = {
'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',,
'referer': 'https://www.amazon.com/',
}
Thanks for any help!!
Solution
You need to uncomment this section in your settings.
DOWNLOADER_MIDDLEWARES = {
'spiders.middlewares.MyProjectDownloaderMiddleware': 543,
}
The name of the class will depend on the name of your project.
Answered By - alexpdev
0 comments:
Post a Comment
Note: Only a member of this blog may post a comment.