[FIXED] Python selenium scraper always returns empty list

Issue

The following code always returns an empty list even though there is clearly an element with the specified class name:

from selenium import webdriver 
from selenium.webdriver.common.by import By 
from selenium.webdriver.chrome.service import Service as ChromeService 
from webdriver_manager.chrome import ChromeDriverManager 
 
# instantiate options 
options = webdriver.ChromeOptions() 
 
# run browser in headless mode 
options.headless = True 
 
# instantiate driver 
driver = webdriver.Chrome(service=ChromeService( 
    ChromeDriverManager().install()), options=options) 
 
# load website 
url = 'https://maclear.ch/en/projects' 
 
# get the entire website content 
driver.get(url) 
 
# select elements by class name 
elements = driver.find_elements(By.CLASS_NAME, 'project-info__elem') 
print(elements)

I would expect to get the specified element

Solution

This seems to be a situation where you are trying to get the list of elements before they are loaded, so naturally you will get an empty list.

Try adding a driver.implicitly_wait(10) after getting the url, and you should get some elements.

Of course you should look into other, more elegant ways to wait for the page to load before you fetch the elements. 10 seconds might not be enough in all cases, and it might be unnessesarily long in other cases.

Answered By - Schpenn