[FIXED] How to deal with cookie pop ups when scraping various websites using Selenium?

Issue

I am trying to scrape ~10k different urls, using Scrapy with Selenium. At some websites, all I can scrape is the cookie pop-up. How to handle those pop-ups in general? Can the selenium driver accept all cookies by default, so that the cookie pop-up would not render? Navigating the pop-ups with selectors seems impossible, as they differ from page to page.

Solution

Different websites are built using different components following different architecture. Some websites may use Vue.js where as some other may use Ember.js.

So in these cases the effective locator strategy to locate the Accept All button on different website is ideally bound to be different.

Hence there can be no generic solution to accept all cookies by default across all websites.

Answered By - undetected Selenium