[FIXED] Inconsistent results retrieving data with Selenium in Python

Issue

driver = webdriver.Chrome(driver_path, options=chrome_options)
wait = WebDriverWait(driver, 20)

driver.get('https://%s/' % asset_id)

wait.until(EC.presence_of_element_located((By.XPATH, "//*[@id='dev_diaginfo_fid']")))

print(driver.find_element_by_xpath('//*[@id='dev_diaginfo_fid']").get_attribute=("innerHTML"))

I'm able to log into the website and Selenium returns the WebElement but it is not consistent when returning the text from that WebElement. Sometimes it returns it and other times it seems like it isn't loading fast enough (slow network where this is being utilized) and returns no data at all but I can still see the WebElement itself just not the data. The data is dynamically loaded via JS. Probably not relevant but I am using send_keys to pass the credentials needed to login and then the page with the version is loaded.

Is there a way to use an ExpectedCondition (EC) to wait until it sees text before moving on? I'm attempting to pull the firmware version from a network device and it finds the Firmware element but it is not consistent when returning the actual firmware version. As stated before, there are issues with network speeds occasionally so my suspicion is that it's moving on before loading the firmware number. This device does not have internet access so I can't share the URL. I can confirm that I have pulled the firmware version but it's just not consistent.

I have tried passing it to beautifulsoup and can verify that it sees Firmware Version: but the inner tags are empty.

Edit: I have tried EC.visibility_of_all_elements and EC.visibility_of_element as well with no luck.

Solution

Here's an idea.

Try a while loop until you see the text.

counter = 0

elem = driver.find_element_by_xpath("//*[@id='dev_diaginfo_fid']").get_attribute=("innerHTML")

while "" in elem:
    pause(500)
    elem = driver.find_element_by_xpath("//*[@id='dev_diaginfo_fid']").get_attribute=("innerHTML")
    if "" not in elem:
        print("Success! The text is: " + elem)
        break
    if counter > 20:
        print("Text still not found!")
        break
    counter += 1

Obviously, adjust the loop to suit your needs.

Answered By - Alichino