[FIXED] How can I scrape a product price from a website using Beautifulsoup?

Issue

I'm trying to extract the latest bid price for this sneaker from StockX but I'm getting the error

IndexError: list index out of range

since sneaker_price is coming up blank for some reason. Can anyone please help?:

from bs4 import BeautifulSoup
from selenium import webdriver

driver = webdriver.Chrome()
driver.get("https://stockx.com/air-jordan-6-retro-travis-scott")

soup = BeautifulSoup(driver.page_source,"lxml")
driver.quit()

sneaker_price = soup.select("div.en-us stat-value stat-small")[0]

Solution

Try CSS selector div.en-us.stat-value.stat-small:

sneaker_price = soup.select("div.en-us.stat-value.stat-small")[0]
print(sneaker_price.text)

Prints:

€523

---

Note: If you get captcha page, try to specify more HTTP headers and/or Cookies. For example:

import requests
from bs4 import BeautifulSoup

headers = {
    'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:80.0) Gecko/20100101 Firefox/80.0',
    'Accept-Language': 'en-US,en;q=0.5'
}

cookies = {
    'stockx_homepage': "sneakers",
}

soup = BeautifulSoup(requests.get("https://stockx.com/air-jordan-6-retro-travis-scott", headers=headers, cookies=cookies).content,"lxml")

sneaker_price = soup.select("div.en-us.stat-value.stat-small")[0]
print(sneaker_price.text)

Answered By - Andrej Kesely