[FIXED] Python - Webscraping - get data from grid and flex field

Issue

I'm using selenium and i can't get data from a DIV marked as flex from https://www.jpg.store/collection/hungrycowsbymuesliswap?tab=items

I need the value from the asset_id attribute (marked in yellow)page source

My code only gets to the div above the value I'm looking for

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.options import Options

chrome_options = Options()
chrome_options.add_argument("--headless")  
chrome_options.add_argument("--disable-gpu")   headless
driver = webdriver.Chrome(options=chrome_options)
url = 'https://www.jpg.store/collection/hungrycowsbymuesliswap?tab=items'
driver.set_page_load_timeout(4) 
driver.get(url)

element = driver.find_element(By.XPATH, "//body/div/div[1]/main/div[2]/div/section/div/div[2]/div/div/div[1]")

print(element.get_attribute('outerHTML'))

if I add another div to xpath, the code will display an error : selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"//body/div/div[1]/main/div[2]/div/section/div/div[2]/div/div/div[1]/div"}

Solution

You can just to search for element array with locator div[asset_id], wait for it's presence, using WebDriverWait and get attribute asset_id for each found element.

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

chrome_options = Options()
chrome_options.add_argument("--headless")
chrome_options.add_argument("--disable-gpu")
driver = webdriver.Chrome(options=chrome_options)
url = 'https://www.jpg.store/collection/hungrycowsbymuesliswap?tab=items'
driver.get(url)
wait = WebDriverWait(driver, 10)
asset_els = wait.until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, "div[asset_id]")))

for element in asset_els:
    print(element.get_attribute('asset_id'))

Answered By - Yaroslavm