Issue
all. I am working on my first Python project and am running into a problem. I am scraping a website and only getting the last result printed out. Nothing seems to work and I have been at it for a day now. here is the link to the site I am trying to scrape: https://www.thomann.de/intl/si/cat_BF_universal_audio.html
and here is my code:
from bs4 import BeautifulSoup
import requests
url = 'https://www.thomann.de/intl/si/cat_BF_universal_audio.html'
page = requests.get(url)
soup = BeautifulSoup(page.content, 'html.parser')
products = soup.find_all('div', class_ = "fx-product-list-entry")
for product in products:
manufacturer = product.find('span', class_= "title__manufacturer").text.strip()
name = product.find('span', class_= "title__name").text.strip()
availability = product.find('div', class_ = 'product__availability product__availability--1')
price = product.find('div', class_ = "fx-price-group product__price-group").text.strip()
Url = product.find('a', class_ = 'product__image')['href']
print(manufacturer, name, price, availability.string.strip(), Url)
Solution
The print function was outside from the for loop
and there was NoneType problem in availability
Tthe url that you are going to scrape is relative url and to make it working I've converted it into absolute url.Here is the full working solution
from bs4 import BeautifulSoup
import requests
url = 'https://www.thomann.de/intl/si/cat_BF_universal_audio.html'
page = requests.get(url)
base_url = 'https://www.thomann.de/intl/si/'
soup = BeautifulSoup(page.content, 'html.parser')
products = soup.find_all('div', class_="fx-product-list-entry")
for product in products:
manufacturer = product.find('span', class_="title__manufacturer").text.strip()
name = product.find('span', class_="title__name").text.strip()
availability = product.find('div', class_='product__availability product__availability--1')
stock = availability.text if availability else None
price = product.find('div', class_="fx-price-group product__price-group").text.strip()
Url = product.find('a', class_='product__image')['href']
abs_url = base_url + Url
print(manufacturer, name, price, stock, abs_url)
Output:
Universal Audio Apollo Twin X Duo Heritage Ed. 1.090 €
In stock
https://www.thomann.de/intl/si/universal_audio_apollo_twin_x_duo_heritage_ed.htm?listPosition=0
Universal Audio Apollo x8p Heritage Edition 3.666 €
In stock
https://www.thomann.de/intl/si/universal_audio_apollo_x8p_heritage_edition.htm?listPosition=1
Universal Audio Apollo x6 Heritage Edition 2.590 €
In stock
https://www.thomann.de/intl/si/universal_audio_apollo_x6_heritage_edition.htm?listPosition=2
Universal Audio UAD-2 Satellite TB3 Octo 1.129 €
In stock
https://www.thomann.de/intl/si/universal_audio_uad_2_satellite_tb3_octo.htm?listPosition=3
Universal Audio Apollo X4 Heritage Edition 2.190 €
In stock
https://www.thomann.de/intl/si/universal_audio_apollo_x4_heritage_edition.htm?listPosition=4
Universal Audio Apollo Twin X Quad Heritage 1.590 €
In stock
https://www.thomann.de/intl/si/universal_audio_apollo_twin_x_quad_heritage.htm?listPosition=5
Universal Audio Apollo Twin USB Duo Heritage 1.090 €
In stock
https://www.thomann.de/intl/si/universal_audio_apollo_twin_usb_duo_heritage.htm?listPosition=6
Universal Audio Apollo x16 Heritage Edition 3.890 €
In stock
https://www.thomann.de/intl/si/universal_audio_apollo_x16_heritage_edition.htm?listPosition=7
Universal Audio OX Amp Top Box 1.269 € None https://www.thomann.de/intl/si/universal_audio_ox_amp_top_box.htm?listPosition=8
Universal Audio Apollo Twin MKII Duo Heritage 990 €
In stock
https://www.thomann.de/intl/si/universal_audio_apollo_twin_mkii_duo_heritage.htm?listPosition=9
Universal Audio Apollo Solo TB3 Heritage Ed. 698 €
In stock
https://www.thomann.de/intl/si/universal_audio_apollo_solo_tb3_heritage_ed.htm?listPosition=10
Universal Audio Solo 610 1.090 €
In stock
https://www.thomann.de/intl/si/universal_audio_solo_610.htm?listPosition=11
Universal Audio UAD-2 Satellite TB3 Octo Cust. 1.590 €
In stock
https://www.thomann.de/intl/si/universal_audio_uad_2_satellite_tb3_octo_cust.htm?listPosition=12
Universal Audio UAFX Starlight Echo Station 398 €
In stock
https://www.thomann.de/intl/si/universal_audio_starlight_delay_pedal.htm?listPosition=13
Universal Audio UAD-2 Satellite USB Octo 1.190 €
In stock
https://www.thomann.de/intl/si/universal_audio_uad_2_satellite_usb_octo.htm?listPosition=14Universal Audio UAFX Golden Reverberator 398 €
In stock
https://www.thomann.de/intl/si/universal_audio_golden_reverberator_pedal.htm?listPosition=15
Universal Audio UAD-2 Satellite TB3 Quad 798 €
In stock
https://www.thomann.de/intl/si/universal_audio_uad_2_satellite_tb3_quad.htm?listPosition=16Universal Audio UAD-2 Live Rack Core 2.990 €
In stock
https://www.thomann.de/intl/si/universal_audio_uad_2_live_rack_core.htm?listPosition=17
Universal Audio UAD-2 Octo 1.190 € None https://www.thomann.de/intl/si/universal_audio_uad_2_octo.htm?listPosition=18
Universal Audio UAFX Astra Modulation Machine 398 €
In stock
https://www.thomann.de/intl/si/universal_audio_astra_modulation_pedal.htm?listPosition=19
Universal Audio 4-710D Twin-Finity 2.590 €
In stock
https://www.thomann.de/intl/si/universal_audio_4710d_twinfinity.htm?listPosition=20
Universal Audio Apollo Thunderbolt 3 Card 198 € None https://www.thomann.de/intl/si/universal_audio_apollo_thunderbolt_3_card.htm?listPosition=21
Universal Audio 710 Twin-Finity 888 € None https://www.thomann.de/intl/si/universal_audio_710_twinfinity.htm?listPosition=22
Universal Audio UAD-2 Sat Octo TB3 Ultimate 9 3.666 €
In stock
https://www.thomann.de/intl/si/universal_audio_uad_2_sat_octo_tb3_ultimate_9.htm?listPosition=23
Universal Audio UAD-2 Satellite USB Quad 798 €
In stock
https://www.thomann.de/intl/si/universal_audio_uad_2_satellite_usb_quad.htm?listPosition=24
Answered By - F.Hoque
0 comments:
Post a Comment
Note: Only a member of this blog may post a comment.