[FIXED] Web Scraping Using Selenium and Beautiful Soup but not Working

Issue

I am trying to get the author and content of the comment from a website but I found out that its Page Source and Inspect Elements are different. I tried to use BeautifulSoup but I cannot get anything back from it. Therefore, I tried to use Selenium but still, I couldn't get anything. I inspect elements from the website and put in the class name by using Selenium but still couldn't scrape anything back. Here is the code that I wrote.

web = "https://www.regulations.gov/document?D=WHD-2020-0007-0609"

#Selenium
driver = webdriver.Chrome()
driver.get(web)
name = driver.find_elements_by_name("GIY1LSJBID")

#Beautifulsoup
page = requests.get(web)
soup = BeautifulSoup(page.text, 'html.parser')
quotes = soup.find_all('div')

I am wondering did I do something wrong and how can I fix it?

Solution

You've already given the answer yourself. You are searching for an element by it's classname, but you use find_elements_by_name. This doesn't search for class name, but the name attribute in an element. Also find_elements with an 's' on the end means the function returns a list of elements not a single element.

In your case you need find_element_by_class_name("GIY1LSJBID")

Answered By - Beek