[FIXED] Why selenium can't find element to click using xpath or css selector?

Issue

I am trying to click on a ">" on the page here but the chrome XPATH or CSS Selector can't find it. I am trying to navigate through dates and get a table per date. https://theanalyst.com/na/2023/08/opta-football-predictions/

Should chrome full xpath just work for selenium?

import sys
import os
import pandas as pd
from selenium import webdriver
from pyvirtualdisplay import Display
from bs4 import BeautifulSoup
from selenium import webdriver
import chromedriver_binary
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By


def parse_website():
    # Start a virtual display
    display = Display(visible=0, size=(800, 600))
    display.start()

    try:
        # Set Chrome options with the binary location
        chrome_options = webdriver.ChromeOptions()
        chrome_options.binary_location = "/usr/bin/google-chrome"

        # Initialize Chrome driver
        driver = webdriver.Chrome()

        # Open the desired URL
        url = "https://theanalyst.com/na/2023/08/opta-football-predictions/"
        driver.get(url)

        # Wait for the page to load completely (adjust the time as needed)
        # Parse the page source using BeautifulSoup
        predictions = WebDriverWait(driver, 10).until(
            EC.presence_of_all_elements_located(
                (By.CSS_SELECTOR, "iframe[src*=predictions]")
            )
        )
        element = driver.find_element(By.CSS_SELECTOR,"#pos_3.div.fixtures-header.button:nth-child(3)")
        element.click()
    except Exception as e:
        print(f"An error occurred: {e}")

This code gets this error:

   An error occurred: Message: no such element: Unable to locate element: {"method":"css selector","selector":"#pos_3.div.fixtures-header.button:nth-child(3)"}

Edit: I needed a wait in the code. Thank you Yaroslavm.

Solution

If I understood you correct, you're trying to navigate through dates and get table output, corresponding to date.

You need to click on last element with locator button.swipe-button inside iframe context.

However, after click you need to wait for new tables are rendered. You can implement wait_for_values_not_equal function and waits for date header content to be changed after click. It seems enough in your case as rendering condition.

Also, you can wait for tables array to be changed after click or even add dirty time.sleep(1) if you don't care about code quality.

I added example, how to get content from first 5 tables and added delimiters for each date.

import time

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By

driver = webdriver.Chrome()
url = 'https://theanalyst.com/na/2023/08/opta-football-predictions/'

def wait_for_values_not_equal(value_func, value, timeout=10, interval=0.3):
    start_time = time.time()

    while value_func() == value:
        if time.time() - start_time > timeout:
            raise TimeoutError("Timeout: Value did not change within the specified time.")
        time.sleep(interval)

    return value_func() != value

def get_date():
    return driver.find_element(By.CSS_SELECTOR, '.fixtures-header h2').text

driver.get(url)
delay = 10
predictions = WebDriverWait(driver, delay).until(
        EC.presence_of_all_elements_located((By.CSS_SELECTOR, 'iframe[src*=predictions]')))

driver.switch_to.frame(predictions[-1])
tables = driver.find_elements(By.CSS_SELECTOR, '[class*=match-card-body] table')
date = None
for i in range(5):
    visible_tables = [table for table in tables if table.is_displayed()]
    print('Date: ')
    date = get_date()
    print(date)
    print('__________________________________________')
    for table in visible_tables:
        rows = table.find_elements(By.CSS_SELECTOR, 'tr')
        for row in rows:
            print(row.text)
    print('__________________________________________')
    swipe_button = driver.find_elements(By.CSS_SELECTOR, 'button.swipe-button')[-1].click()
    wait_for_values_not_equal(get_date, date)
    tables = WebDriverWait(driver, delay).until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, '[class*=match-card-body] table')))

Answered By - Yaroslavm