[FIXED] Why do I get an empty list when I web-scrape customer reviews from Amazon(using BeautifulSoup)?

Issue

I'm trying to web-scrape customer reviews from an Amazon page for a data science project. Everytime I run this following piece of code, I receive an empty list.

from bs4 import BeautifulSoup 
import requests
import pandas as pd


url = "https://www.amazon.in/Nike-Phantom-White-Light-Cream-Particle-Running/dp/B07HYZ2Q71/ref=sr_1_14?keywords=nike%2Brunning%2Bshoes%2Bfor%2Bwomen&qid=1662552658&sprefix=nike%2Brunning%2Caps%2C207&sr=8-14&th=1"

htmlContent = requests.get(url)

soup = BeautifulSoup(htmlContent.content, "html.parser")

review_titles = soup.find_all('div', {"data-hook" : "review-title"})
print(review_titles)

Where am I going wrong? I don't know much about web-scraping.

Solution

try:

url = "https://www.amazon.in/Nike-Phantom-White-Light-Cream-Particle-Running/dp/B07HYZ2Q71/ref=sr_1_14?keywords=nike%2Brunning%2Bshoes%2Bfor%2Bwomen&qid=1662552658&sprefix=nike%2Brunning%2Caps%2C207&sr=8-14&th=1"

htmlContent = requests.get(url)

soup = BeautifulSoup(htmlContent.content, "html.parser")

for review in soup.find('div', {'data-hook': 'top-customer-reviews-widget'}).find_all('div', {'data-hook': 'review'}):
    result = {
        'name': review.find('span', class_='a-profile-name').get_text().strip(),
        'title': review.find('a', {'data-hook': 'review-title'}).get_text().strip(),
        'stars': review.find('a', class_='a-link-normal').get('title').strip(),
        'data': review.find('span', {'data-hook': 'review-date'}).get_text().strip(),
        'text': review.find('div', {'data-hook': 'review-collapsed'}).get_text().strip()
    }
    print(result)

OUTPUT:

{'name': 'The Sidturbing Element', 'title': 'I guess I was unlucky!', 'stars': '3.0 out of 5 stars', 'data': 'Reviewed in India on 28 August 2019', 'text': 'The product was delivered in only a plain almost tattered nike box. There was no effort put in packaging as such. The shoe was too small. I bought a UK8 but ittl felt like a 7 or a 6.5. Felt sturdy but somehow lacked in the flexibility. Made me doubt if the product was original.'}
{'name': 'Padma namgyal', 'title': 'Good', 'stars': '4.0 out of 5 stars', 'data': 'Reviewed in India on 5 October 2019', 'text': 'Nice but it seems like 1st copy...'}
{'name': 'Suresh', 'title': 'Major design failure.', 'stars': '1.0 out of 5 stars', 'data': 'Reviewed in India on 4 May 2021', 'text': 'Although the size was correct but my foot was not going in the shoes I tried to loosen the lace but still I was not able to put on the shoes. I believe it’s a design issue,the shoes were made in Indonesia and I think it’s a major disaster from a company like Nike. Please look into this very seriously and inform the company regarding this so that they can take appropriate steps.'}
{'name': 'Vijay Bhaskar. M', 'title': "If it's cloud tail then buy!", 'stars': '5.0 out of 5 stars', 'data': 'Reviewed in India on 30 July 2019', 'text': 'I like cloud tail vendor,  lots of love to this enterprise!  Please notify me about any other shoes.  I will purchase only from you'}
{'name': 'Clark Kent', 'title': 'Faulty design ruins everything....too tight....', 'stars': '2.0 out of 5 stars', 'data': 'Reviewed in India on 15 May 2019', 'text': 'The material quality is best. No doubt about NIKE. But the design has a flaw. Even if you order your right size, the shoes are tight. So tight that it makes it difficult to wear even for a short time.  This is the example where faulty design ruins the use if good materials..... And I am not sure that ordering one size higher would help.....I would not recommended.'}
{'name': 'Anil Kumar', 'title': 'Not Good.', 'stars': '3.0 out of 5 stars', 'data': 'Reviewed in India on 11 November 2019', 'text': 'Size fit was not right -'}
{'name': 'Raviraj.V.R', 'title': 'Product genuiness', 'stars': '3.0 out of 5 stars', 'data': 'Reviewed in India on 24 July 2019', 'text': 'M not too sure about the brands genuinity'}
{'name': 'P jyothi', 'title': 'Nice product', 'stars': '4.0 out of 5 stars', 'data': 'Reviewed in India on 19 February 2019', 'text': 'Good looking'}

Answered By - Sergey K