[FIXED] Python web scraping: How to combine selenium and pandas for gathering data on HTML?

Issue

I am gathering sports fixtures and results on the webpage, first of all, I am going to use Pandas to scrape, however, there is an option for selecting "timezone" on the page, so I add slenium for the auto-choosing timezone, therefore I do not know how to scrape with pandas after I use slenium. Would everybody please do me a favour, thank you very much.

here is my work:

from selenium import webdriver
from selenium.webdriver.support.ui import Select
import pandas as pd

PATH ="C:/Users/XXX/Desktop/chromedriver.exe"
driver = webdriver.Chrome( PATH )

driver.get("https://fixturedownload.com")

select = Select(driver.find_element_by_name("timezone"))

select.select_by_value("SE Asia Standard Time" )

driver.find_element_by_xpath('/html/body/div[2]/div/div[2]/form/div/input[1]').click()

List = pd.read_html(I am stuck here)

Solution

To select the timezone as SE Asia Standard Time and scrape the TABLE using Pandas you can use the following Locator Strategies:

Code Block:

from selenium import webdriver
from selenium.webdriver.support.ui import Select
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
import pandas as pd

driver.get("https://fixturedownload.com/")
Select(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//select[@name='timezone']")))).select_by_value("SE Asia Standard Time" )
driver.find_element(By.XPATH, "//input[@value='Set Timezone']").click()
data = WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//table[@class='fixture']"))).get_attribute("outerHTML")
df  = pd.read_html(data)
print(df)

Console Output:

[                    0                1  ...                          4          5
0        Full fixture  Preview fixture  ...  Download fixture for ICAL  View JSON
1               Teams            Teams  ...                      Teams        NaN
2      Adelaide Crows  Preview fixture  ...  Download fixture for ICAL  View JSON
3      Brisbane Lions  Preview fixture  ...  Download fixture for ICAL  View JSON
4             Carlton  Preview fixture  ...  Download fixture for ICAL  View JSON
5         Collingwood  Preview fixture  ...  Download fixture for ICAL  View JSON
6            Essendon  Preview fixture  ...  Download fixture for ICAL  View JSON
7           Fremantle  Preview fixture  ...  Download fixture for ICAL  View JSON
8        Geelong Cats  Preview fixture  ...  Download fixture for ICAL  View JSON
9     Gold Coast Suns  Preview fixture  ...  Download fixture for ICAL  View JSON
10         GWS Giants  Preview fixture  ...  Download fixture for ICAL  View JSON
11           Hawthorn  Preview fixture  ...  Download fixture for ICAL  View JSON
12          Melbourne  Preview fixture  ...  Download fixture for ICAL  View JSON
13    North Melbourne  Preview fixture  ...  Download fixture for ICAL  View JSON
14      Port Adelaide  Preview fixture  ...  Download fixture for ICAL  View JSON
15           Richmond  Preview fixture  ...  Download fixture for ICAL  View JSON
16           St Kilda  Preview fixture  ...  Download fixture for ICAL  View JSON
17       Sydney Swans  Preview fixture  ...  Download fixture for ICAL  View JSON
18  West Coast Eagles  Preview fixture  ...  Download fixture for ICAL  View JSON
19   Western Bulldogs  Preview fixture  ...  Download fixture for ICAL  View JSON

[20 rows x 6 columns]]

Answered By - undetected Selenium