Issue
Edit: Got the answer. The autothrottle was limited to default. Now I need to limit it to the number of website pages. The code looks like this CONCURRENT_REQUESTS = 3
I am using scrapy playwright. I want to loop through some sites. The scrapy spider will request to extract data through each website link. After rendering through each request, do I need to close the tab? like using this:
def parse(self, response):
page = response.meta["playwright_page"]
await page.close()
Solution
Edit: Got the answer. The autothrottle was limited to default. Now I need to limit it to the number of website pages. The code looks like this CONCURRENT_REQUESTS = 3
Answered By - Shahidul Islam Pranto
0 comments:
Post a Comment
Note: Only a member of this blog may post a comment.