[FIXED] Hover the Mouse to the tab and scrape the products

Issue

Hello I am trying to use selenium to automatically scrape the products titles and prices, i am using ActionChains and move_to_element, but somehow it gave me timeout exception, Is there a better way to do it? titles in the tab

https://denago.com/collections/ebikes

#For Dynamic webpage, import selenium
!apt-get update
!apt install chromium-chromedriver
!cp /usr/lib/chromium-browser/chromedriver /usr/bin
!pip install selenium
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.action_chains import ActionChains
from selenium.webdriver.support.ui import WebDriverWait 
from selenium.webdriver.support import expected_conditions as EC
#set up Chrome driver
options=webdriver.ChromeOptions()
options.add_argument('--headless')
options.add_argument('--no-sandbox')
options.add_argument('--disable-dev-shm-usage')
#Define web driver as a Chrome driver
driver=webdriver.Chrome('chromedriver',options=options)
driver.implicitly_wait(10)



driver.get('https://denago.com/collections/ebikes')
action = ActionChains(driver)
ourbike = WebDriverWait(driver, 5).until(EC.visibility_of_element_located((By.XPATH, "/html/body/div[6]/div/header/nav/ul/li[1]/a/span")))
ActionChains(driver).move_to_element(ourbike).perform()
Titles=driver.find_elements(By.CLASS_NAME,'mm-title')
for i in range(len(Titles)):
  print(Titles[i].text)

Solution

There are a couple of problems:

1. Browser opens at the default size which is small and the element you are searching for to hover is absent on the page. So you need to set options.add_argument('window-size=1200,1980').
2. There is a message about cookies that overlaps elements on the page. It's better to close it: driver.find_element(By.ID, 'CybotCookiebotDialogBodyLevelButtonLevelOptinAllowAll').click()
3. The element you tried to hover could not be found by the XPATH you used. It can easily be found with (By.XPATH, '(//li[@itemid="m9RVB"])'), but there are two such elements on the page and the first is hidden. So you need to hover the second one, so add [2] to the locator: (By.XPATH, '(//li[@itemid="m9RVB"])[2]')
   So, here is the code:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.action_chains import ActionChains
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

#set up Chrome driver
options = webdriver.ChromeOptions()
options.add_argument('--headless')
options.add_argument('--no-sandbox')
options.add_argument('--disable-dev-shm-usage')
options.add_argument('window-size=1200,1980')
#Define web driver as a Chrome driver
driver = webdriver.Chrome('chromedriver', options=options)
driver.implicitly_wait(10)

driver.get('https://denago.com/collections/ebikes')
driver.find_element(By.ID, 'CybotCookiebotDialogBodyLevelButtonLevelOptinAllowAll').click()
action = ActionChains(driver)
ourbike = WebDriverWait(driver, 5).until(EC.visibility_of_element_located((By.XPATH, '(//li[@itemid="m9RVB"])[2]')))
ActionChains(driver).move_to_element(ourbike).perform()
Titles = driver.find_elements(By.CLASS_NAME, 'mm-title')
for i in range(len(Titles)):
    print(Titles[i].text)
driver.quit()

Answered By - Eugeny Okulik