Issue
I was crawling a website when they blocked me with a 403. I can still access the website from any of my browser but my Scrapy script is unable to crawl.
I crawl from home using 5 proxies and each has 10 IPs randomly selected. I have about 40 user_agents selected randomly (see code below). A new set of proxy and the user_agent is selected on each request. Each yield request is about 1MB heavy. When about 100 requests are collected the script (using S3pipelines from Github) sends a package to my S3 on AWS. I have a download delay of 10 and auto throttles activated in the settings of my Scrapy project. All that is aligned with the website robots.txt request -crawling with delay 10.
I read instructions manuals from Scrapy, from Python, and some others but couldn't find any way to become more 'human' in crawling.
My settings (extract):
BOT_NAME = 'acCrawler10'
SPIDER_MODULES = ['acCrawler10.spiders']
NEWSPIDER_MODULE = 'acCrawler10.spiders'
s3 = boto3.client('s3', aws_access_key_id='AWS_ACCESS_KEY_ID',
aws_secret_access_key='AWS_SECRET_ACCESS_KEY')
USER_AGENT = get_random_agent()
# Obey robots.txt rules
ROBOTSTXT_OBEY = True
CONCURRENT_REQUESTS = 32
DOWNLOAD_DELAY = 10
PROXYMESH_TIMEOUT = 60
# CONCURRENT_REQUESTS_PER_DOMAIN = 16
CONCURRENT_REQUESTS_PER_IP = 16
# Disable cookies (enabled by default)
# COOKIES_ENABLED = False
AUTOTHROTTLE_ENABLED = True
AUTOTHROTTLE_START_DELAY = 5
AUTOTHROTTLE_MAX_DELAY = 60
AUTOTHROTTLE_TARGET_CONCURRENCY = 4.0
AUTOTHROTTLE_DEBUG = False
Solution
I used the settings from Freddy with a small modification and as suggested from @Agus-Mathew reducing the "CONCURRENT_REQUESTS_PER_IP".
So, >my< final settings that did the job was:
DOWNLOAD_DELAY = 10
CONCURRENT_REQUESTS = 1
...
#PROXYMESH_TIMEOUT = 60
#CONCURRENT_REQUESTS_PER_IP = 16
#Disable cookies (enabled by default)
#COOKIES_ENABLED = False
...
AUTOTHROTTLE_ENABLED = True
AUTOTHROTTLE_START_DELAY = 5
AUTOTHROTTLE_MAX_DELAY = 60
AUTOTHROTTLE_TARGET_CONCURRENCY = 4.0
#AUTOTHROTTLE_DEBUG = False
I configured my Scrapy header as follows, because I received the block only with my Scrapy script - but not with my web browser (like Freddy). Rotating proxies I didn't use for the job.
DEFAULT_REQUEST_HEADERS = {
'accept-encoding': 'gzip, deflate, br',
'accept-language': 'en-US,en;q=0.9',
'sec-ch-ua': '"Chromium";v="106", "Microsoft Edge";v="106", ";Not A Brand";v="99"',
'sec-ch-ua-mobile': '?0',
'sec-ch-ua-platform': '"Windows"',
}
But of course - it depends on the website we are trying to scrape from. If their API settings are overly sensitive to requests, then we must adjust accordingly.
By the way: Often I read about using the "time.seep()" (or individual not using, cause of Twisted Reactor and the Scrapy concurrency) in the spider. But in my case time.sleep was without any effect avoiding the error. So it doesn't help me.
Answered By - Legion Inc.
0 comments:
Post a Comment
Note: Only a member of this blog may post a comment.