[FIXED] BeautifulSoup <small> tag

Issue

I want to scrape the text between <> element. HTML code where I want to scrape "text" inside small (which is was: 27.00). The HTML is <> class="product-views-price-old" Was: £27.00 <> My code is:

from bs4 import BeautifulSoup
import requests
url = "https://www.petshop.co.uk/Dog"
r = requests.get(url)
soup = BeautifulSoup(r.content)
for old_price in soup.find_all("small", class_ = "product-views-price-old"):
    print(old_price)

The above code gives me nothing. Even no error. How can I scrape the text between <> tags?

Solution

Content is served dynamically, so you wont get it this way with requests - Take a look at this selenium code.

To get rid of text and spaces you can do:

.get_text(strip=True).replace('Was: ','')

Example

from selenium import webdriver
from bs4 import BeautifulSoup
import time

url = "https://www.petshop.co.uk/Dog"
driver = webdriver.Chrome('C:\Program Files\ChromeDriver\chromedriver.exe')
driver.get(url)
time.sleep(3)

html = driver.page_source
soup = BeautifulSoup(html,'html.parser')
for old_price in soup.find_all("small", class_ = "product-views-price-old"):
    print(old_price.get_text(strip=True).replace('Was: ',''))

driver.quit()

Output

£2.20
£18.61
£27.00
£38.39
£38.39
£20.65
£1.30
£67.99
£20.65
£1.30
£54.95
£30.99

Answered By - HedgeHog