Issue
I'm really new to this and am confused on how I can select information with a particular div class image of html but it is <div class="row" data-start="1.."). I don't understand what data-start is considered within the and how I can get the pricing information from that table, it just keeps showing me none as an output. Any help would be much appreciated.
here is my code:
from bs4 import BeautifulSoup
import bs4
import requests
import json
import numpy as np
urls = ['https://filterbuy.com/brand/trion-air-bear-air-filters/20x20x5-air-bear-20x20/?selected_merv=8',
'https://filterbuy.com/brand/trion-air-bear-air-filters/20x20x5-air-bear-20x20/?selected_merv=11',
'https://filterbuy.com/brand/trion-air-bear-air-filters/20x20x5-air-bear-20x20/?selected_merv=13']
#scrape elements
for url in urls:
response = requests.get(url)
soup = BeautifulSoup(response.content, "html.parser")
#returned values
product = soup.find("h1", class_="text-center")
merv = url.rsplit('?', 0)
price_table= soup.find("??????")
json_schema = soup.find_all('script', attrs={'type': 'application/ld+json'})[1]
json_file = json.loads(json_schema.get_text())
for product, merv, in zip(product, merv):
print(product.getText(), merv, price_table, json_file)
np.savetxt('products.csv', [p for p in zip(product, json_schema)], delimiter=',', fmt='%s')
Solution
The problem is that this table is not in the source code, it is formed from static data using a JS. All three tables exist immediately on the page. You can access tax values in the cart class
url = 'https://filterbuy.com/brand/trion-air-bear-air-filters/20x20x5-air-bear-20x20/?selected_merv=8'
response = requests.get(url)
mervs = BeautifulSoup(response.text, 'lxml').find_all('strong')
for i, cart in enumerate(BeautifulSoup(response.text, 'lxml').find_all('form', class_='cart')):
for tax in cart.attrs:
if 'data-price' in tax:
print(mervs[i].get_text(), cart[tax])
OUTPUT:
MERV 8 47.58
MERV 8 29.73
MERV 8 28.11
MERV 8 27.56
MERV 8 27.03
MERV 11 57.09
MERV 11 29.13
MERV 11 30.92
MERV 11 30.32
MERV 11 30.02
MERV 11 29.73
MERV 13 62.99
MERV 13 34.11
MERV 13 32.54
MERV 13 32.01
MERV 13 31.49
Answered By - Sergey K
0 comments:
Post a Comment
Note: Only a member of this blog may post a comment.