[FIXED] Selenium with Python / Navigating to next page

Issue

I have a hard time browsing through the 448 consecutive pages of the following page https://www.digitalwallonia.be/fr/cartographie/ with Selenium under Python in a robust manner. I tried (too) many things without satisfactory result (hence, difficult to put relevant code).

Would like to see your solution. Apologize if the question is not appropriately formulated: first timer.

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException

browser = webdriver.Firefox()
browser.implicitly_wait(20)


browser.get('https://www.digitalwallonia.be/fr/cartographie')
browser.find_element("xpath",'//*[@id="axeptio_btn_acceptAll"]').click()
browser.find_element("xpath",'//*[@id="axeptio_btn_configure"]').click()
browser.find_element("xpath",'//*[@id="axeptio_btn_acceptAllAndNext"]').click()

WebDriverWait(browser, 1000).until(EC.element_to_be_clickable((By.CLASS_NAME,'next'))).click()

input('Press ENTER to close the automated browser')
browser.quit()

I get the following error: selenium.common.exceptions.ElementNotInteractableException: Message: Element could not be scrolled into view

Solution

I would advice here about several issues:

1. You should preferably use WebDriverWait, not implicitly_wait since the former is waiting for element presence only while with WebDriverWait you can wait for more mature element states i.e. to be visible, clickable and more.
2. Don't mix WebDriverWait and implicitly_wait in the same file, it may cause problems.
3. The next page buttons are on the bottom of the page, so you will need to scrool down and only after that to click the pager button.
4. No need to set the timeout for more than 30 seconds.
   The code below is working:

import time

from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.action_chains import ActionChains
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC


options = Options()
options.add_argument("start-maximized")


webdriver_service = Service('C:\webdrivers\chromedriver.exe')
driver = webdriver.Chrome(service=webdriver_service, options=options)
url = "https://www.digitalwallonia.be/fr/cartographie"
actions = ActionChains(driver)

wait = WebDriverWait(driver, 10)
driver.get(url)

wait.until(EC.element_to_be_clickable((By.XPATH, '//*[@id="axeptio_btn_acceptAll"]'))).click()
wait.until(EC.element_to_be_clickable((By.XPATH, '//*[@id="axeptio_btn_configure"]'))).click()
wait.until(EC.element_to_be_clickable((By.XPATH, '//*[@id="axeptio_btn_acceptAllAndNext"]'))).click()

driver.execute_script("window.scrollBy(0, arguments[0]);", 800)
time.sleep(0.5)
wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, '.next a'))).click()

Answered By - Prophet