[FIXED] Web Scraping by Using BeautifulSoup in python

Issue

How can I use the json module to extract the price from provides the data in JSON format in an inline script?

I tried to extract the price in https://glomark.lk/top-crust-bread/p/13676 But I couldn't to get the price value.

So please help me to solve this.

import requests
import json

import sys
sys.path.insert(0,'bs4.zip')
from bs4 import BeautifulSoup

user_agent = {
                 'User-agent': 'Mozilla/5.0 Chrome/35.0.1916.47'
                 }
headers = user_agent

url = 'https://glomark.lk/top-crust-bread/p/13676'
req = requests.get(url, headers = headers)
soup = BeautifulSoup(req.content, 'html.parser')

products = soup.find_all("div", class_ = "details col-12 col-sm-12 
col-md-6 col-lg-5 col-xl-5")
for product in products:
    product_name = product.h1.text
    product_price = product.find(id = 'product-promotion-price').text
    print(product_name)
    print(product_price)

Solution

You can grab json data(price) from hidden api using only requests module. But the product name is not dynamic.

import requests
headers= {
    'content-type': 'application/json',
    'x-requested-with': 'XMLHttpRequest'
   }

api_url = "https://glomark.lk/product-page/variation-detail/13676"


jsonData = requests.post(api_url,  headers=headers).json()

price=jsonData['price']
print(price)

Output:

95

Full working code:

from bs4 import BeautifulSoup
import requests
headers= {
    'content-type': 'application/json',
    'x-requested-with': 'XMLHttpRequest'
   }

api_url = "https://glomark.lk/product-page/variation-detail/13676"


jsonData = requests.post(api_url,  headers=headers).json()

price=jsonData['price']



#to grab product name(not dynamic)

url = 'https://glomark.lk/top-crust-bread/p/13676'
req = requests.get(url)
soup = BeautifulSoup(req.content, 'html.parser')

title=soup.select_one('.product-title h1').text
print(title)
print(price)


 

Output:

Top Crust Bread
95
     

Answered By - F.Hoque