[FIXED] Why can't selenium find anything within this div?

Issue

Selenium can't find anything past <div id="content">. Even though there are multiple layers underneath that div. Including the div that I need to get data from.

To try and directly get the elements I am interested in I tried this:

departures = driver.find_elements_by_class_name('departure')
    for departure in departures:
        print(departure)

When departures kept showing up as an empty list I started looking at which point selenium wasn't able to find anything. And discovered that the last div it can find is <div id="content">

then I tried to get the innerHTML of that div:

content = driver.find_element_by_xpath('//*[@id="content"]')
print(content)
HTML = content.get_attribute("innerHTML")
print(HTML)

HTML is an empty string, suggesting that there is nothing within <div id="content">.

Important HTML part: https://pastebin.com/L5Giz0H0

In the HTML part you can see that <div id="content"><div id="OutboundDepartures" class="timetable"> is one line. But when inspecting the page in Firefox <div id="OutboundDepartures" class="timetable"> does appear as a sub-div within <div id="content">

Complete HTML: https://pastebin.com/h07UpdqM

How can I get the data from the <div class="departure"> div when I can't get past <div id="content">?

Solution

Use WebDriverWait and CSS Selctor to get all info.

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By

wait = WebDriverWait(driver, 20)
items=wait.until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, '#content div.departure')))

for item in items:
    print(item.text)

Output:

08:30 Sneldienst
reisduur ca 45 min. 0 meter vrij 353 pers. vrij
09:45 Veerdienst Ms. Friesland
reisduur ca 120 min. 297 meter vrij 678 pers. vrij
12:30 Sneldienst
reisduur ca 45 min. 0 meter vrij 322 pers. vrij
15:00 Veerdienst Ms. Friesland
reisduur ca 120 min. 175 meter vrij 708 pers. vrij
17:20 Sneldienst
reisduur ca 45 min. 0 meter vrij 365 pers. vrij
19:55 Veerdienst Ms. Friesland
reisduur ca 120 min. 196 meter vrij 731 pers. vrij

Answered By - KunduK