Issue
I am trying to web scrape product reviews from a page but I'm not sure how to extract a var inside the <script> tags.
Here's my python code:
import requests
from bs4 import BeautifulSoup
import csv
a_file = open("ProductReviews.csv", "a")
writer = csv.writer(a_file)
# Write the titles of the columns to the CSV file
writer.writerow(["created_at", "reviewer_name", "rating", "content", "source"])
url = 'https://www.lazada.com.my/products/iron-gym-total-upper-body-workout-bar-i467342383.html'
# Connect to the URL
response = requests.get(url)
# Parse HTML and save to BeautifulSoup object
soup = BeautifulSoup(response.content, "html.parser")
data = soup.findAll('script')[123]
if 'var __moduleData__' in data.string:
print("Yes")
Here's the page source (I removed the unnecessary code):
<html>
<head>
<title></title>
</head>
<body>
<script>
var __moduleData__ = {
"data": {
"root": {
"fields": {
"review": {
"reviews": [{
"rating": 5,
"reviewContent": "tq barang dah sampai",
"reviewTime": "24 May 2021",
"reviewer": "Jaharinbaharin",
}, {
"rating": 5,
"reviewContent": "Beautiful quality👌👌👌",
"reviewTime": "08 Sep 2021",
"reviewer": "M***.",
}, {
"rating": 5,
"reviewContent": "the box was badly dented but the item was intact...just that my door frame is shallow and slippery....I can't pull up without worrying of falling down",
"reviewTime": "25 Aug 2021",
"reviewer": "David S.",
}, {
"rating": 5,
"reviewContent": "Haven’t really opened it yet but please put some effort on the packaging for future improvement thanks it was really fast",
"reviewTime": "14 Dec 2020",
"reviewer": "Yasir A.",
}, {
"rating": 5,
"reviewContent": "Seems to be ok, good quality.. No weight restriction mentioned on the box.. I'm about 90kg, it could handle my weight so far..",
"reviewTime": "22 May 2020",
"reviewer": "Kevin",
}]
},
}
},
},
};
</script>
</body>
</html>
I just want to get the review data only so I'd like to know how to extract the value of var __moduleData__.
Solution
You can use a regex to select your variable:
json.loads(re.search(r'var __moduleData__ = ({.*})', response.text).group(1))
Example
from bs4 import BeautifulSoup
import json,re,requests
url = 'https://www.lazada.com.my/products/iron-gym-total-upper-body-workout-bar-i467342383.html'
response = requests.get(url)
d = json.loads(re.search(r'var __moduleData__ = ({.*})', response.text).group(1))
d['data']['root']['fields']['seller']
Output
{'chatResponsiveRate': {'labelText': 'Chat Response', 'value': '100%'},
'chatUrl': 'https://pages.lazada.com.my/wow/i/my/im/chat?brandId=21411',
'hideAllMetrics': False,
'imEnable': True,
'imUserId': '100285367',
'name': 'MR SIX PACK',
'newSeller': False,
'percentRate': '96%',
'positiveSellerRating': {'labelText': 'Seller Ratings', 'value': '96%'},
'rate': 0.96,
'rateLevel': 3,
'sellerId': '1000052649',
'shipOnTime': {'labelText': 'Ship On Time', 'value': '97%'},
'shopId': 255007,
'size': 5,
'time': 2,
'type': '4',
'unit': 'years',
'url': '//www.lazada.com.my/shop/mr-six-pack/?itemId=467342383&channelSource=pdp'}
Answered By - HedgeHog
0 comments:
Post a Comment
Note: Only a member of this blog may post a comment.