[FIXED] How to fix error driver.find_element(By.CLASS_NAME, name), and save the result at txt file?

Issue

This is the HTML of the results of any search at virustotal.

<div class="positives"> <!--?lit$32274217$-->39 </div>

HTML image of the website and the needed data to be extracted

My Code

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import time

driver = webdriver.Chrome()
driver.get("https://www.virustotal.com/gui/home/search")
driver.maximize_window()

wait = WebDriverWait(driver, 15)
search_input = wait.until(EC.element_to_be_clickable((driver.execute_script("return document.querySelector('home-view').shadowRoot.querySelector('vt-ui-search-bar').shadowRoot.querySelector('input#searchInput')"))))
search_input.send_keys("759b1fcbdf4ad54bb8adc2c060df8ab5")
search_input.send_keys(Keys.RETURN)  # Press Enter
time.sleep(5)

# Get the result and write it to a file
result = driver.find_element(By.CLASS_NAME, 'div > div > div.positives')
result_text = str(result.text)
with open('output.txt', 'a') as f:
    f.write(result_text)
    f.write('\n' * 10)

driver.quit()

Error:

 result = driver.find_element(By.CLASS_NAME, 'div > div > div.positives')
 in find_element
    return self.execute(Command.FIND_ELEMENT, {"using": by, "value": value})["value"]

Solution

Root cause of the error:

result = driver.find_element(By.CLASS_NAME, 'div > div > div.positives')

Above line is incorrect. It should be By.CSS_SELECTOR, 'div > div > div.positives'

Solution: Having said the above, just the same way as in this answer, you need to handle shadow-root elements.

Change the code as below:

result = wait.until(EC.element_to_be_clickable((driver.execute_script("return document.querySelector('file-view').shadowRoot.querySelector('vt-ui-main-generic-report').shadowRoot.querySelector('vt-ui-detections-widget').shadowRoot.querySelector('div.positives')"))))
# result = driver.find_element(By.CLASS_NAME, 'div > div > div.positives')

UPDATE:

result = wait.until(EC.element_to_be_clickable((driver.execute_script("return document.querySelector('file-view').shadowRoot.querySelector('vt-ui-main-generic-report').shadowRoot.querySelector('vt-ui-detections-widget').shadowRoot.querySelector('div.positives')"))))
# result = driver.find_element(By.CSS_SELECTOR, 'positives')
result_text = str(result.text)
print(result_text)
with open('output.txt', 'a') as f:
    f.write(result_text)
    f.write('\n' * 10)

driver.quit()

Console result:

39

Process finished with exit code 0

Answered By - Shawn