[FIXED] Can not access / click Button with Selenium

Issue

On the following Website Forge, I need to access the Load more button (at the bottom of the component list) multiple times, to scrape the list of available components. While I am able to access any element that I need, I struggle to access the button directly and then click it.

When using

button = browser.find_element(By.XPATH, '//*[@id="PortalTheme_wt778_block_wtMainContent_wtLoadMore"]')
button.click()

I get the element. However, I cannot click it.

I have no experience in web development, and I am new to web scraping. Thus, I am struggling to find out what exactly is going wrong here.

I would appreciate it if someone could tell me how to click this button!

Thanks in advance :)

Solution

You need to wait until Load More is ready to be clicked. Use Selenium's Waits to perform action on the desired button.

Refer code below which will use Explicit Waits to click Load More:

import time
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By

driver =  webdriver.Chrome()
driver.get("https://www.outsystems.com/forge/list?q=&t=&o=most-popular&tr=False&oss=False&c=%205361,5362,5363,5364,5365,5366,5367,5368,5369,5370,5381,5382,5383,5384,5385,5386,5387,5388,5389,5390,5391,3485,5392,5393&a=&v=&hd=False&tn=&scat=forge")
driver.maximize_window()

# Create wait object to wait for 20s
wait = WebDriverWait(driver,20)

# Click on Accept cookies button
wait.until(EC.element_to_be_clickable((By.XPATH, "//button[@id='onetrust-accept-btn-handler']"))).click()

# Click on Load more button
wait.until(EC.element_to_be_clickable((By.XPATH, "//*[@id='PortalTheme_wt778_block_wtMainContent_wtLoadMore']"))).click()
time.sleep(10)

Answered By - Shawn