[FIXED] Cannot retrieve text from website with selenium for python

Issue

i am trying to make a little program that gets the price of electricity each hour. it will retrieve the text displaying the price of the electricity.

Here is my code

from selenium import webdriver
from selenium.webdriver.common.by import By

option = webdriver.ChromeOptions()
option.add_argument('-incognito')
option.add_argument("--headless")
option.add_argument('--enable-javascript')
option.add_argument("disable-gpu")

browser = webdriver.Chrome(executable_path='chromedriver.exe', options=option)
browser.get('https://www.elektrikell.ee/')

hind = browser.find_element(by=By.XPATH, value='//*[@id="root"]/div/div/div[1]/div[2]/div[2]/b')

Once i run it, it loads up the Webdriver and then crashes with the following error

Message: no such element: Unable to locate element: {"method":"xpath","selector":"//*[@id="root"]/div/div/div[1]/div[2]/div[2]/b"}
  (Session info: headless chrome=103.0.5060.134)

it happens to everything except the HTML , when i copy that Xpath, it works fine. I enabled Javascript because i thought it might fix it, yet here i am.

Does python not support JavaScript based websites?

Solution

To extract the price text ideally you need to induce WebDriverWait for the visibility_of_element_located() and you can use either of the following locator strategies:

• Using CSS_SELECTOR:

  driver.execute("get", {'url': 'https://www.elektrikell.ee/'})
  print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "div.priceHeader div.priceElement"))).text)
  

• Using XPATH:

  driver.execute("get", {'url': 'https://www.elektrikell.ee/'})
  print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//div[@class='priceHeader']//div[@class='priceElement']"))).text)
  

• Console output:

  37.81
  senti / kilovatt-tund
  

• Note : You have to add the following imports :

  from selenium.webdriver.support.ui import WebDriverWait
  from selenium.webdriver.common.by import By
  from selenium.webdriver.support import expected_conditions as EC
  

You can find a relevant discussion in How to retrieve the text of a WebElement using Selenium - Python

Answered By - undetected Selenium