[FIXED] Webscraping question in python using Selenium

Issue

I am trying to scrape using selenium in python. I want the solar data from this site and section: https://www.caiso.com/TodaysOutlook/Pages/supply.html#section-renewables-trend

I think the problem I'm having is that the Chart data (CSV) menu option does not function as a button so clicking it doesn't work. This is what I see when I inspect the element before and after clicking it the "Chart data (CSV)" menu option.

Before: <a class="dropdown-item mb-0" id="downloadRenewablesCSV" data-type="text/csv">Chart data (CSV)</a>

After: <a class="dropdown-item mb-0" id="downloadRenewablesCSV" data-type="text/csv" href="data:text/csv;charset=utf8,Renewables%2007%2F20%2 ... [alot of encoded data] ...2C209%2C211%2C211%2C211%2C212%2C211%2C211%2C210%0A" download="CAISO-renewables-20220720.csv">Chart data (CSV)</a>

originally I assumed it was just a button element that would download the csv file and was trying to do this:

from selenium import webdriver
from selenium.webdriver.common.keys import Keys

driver = webdriver.Chrome(executable_path='PATH')
driver.get('https://www.caiso.com/TodaysOutlook/Pages/supply.html')
button = driver.find_element(by='xpath',value='/html/body/div[1]/div[3]/div[8]/div/div/div[2]/nav/div[3]/div/a[1]')
button.click()

This isn't working. Any advice? I am very new to selenium sorry.

Solution

JS Path Interaction:

Xpath selectors can be a bit finicky, I would revert to the basics and try to interact with the element via the JS Path. I was able to reproduce the error and download the report using the JS Path instead. Implement the following updated code:

driver.get('https://www.caiso.com/TodaysOutlook/Pages/supply.html')
driver.execute_script("el = document.querySelector('#downloadRenewablesCSV');el.click();")

Answered By - Luke Hamilton