[FIXED] Trying to scrape Aliexpress product reviews with Beautifulsoup

Issue

I'm trying to collect aliexpress reviews from a product page e.g https://www.aliexpress.com/item/3256801798731854.html

I have written my code that will scrape this page & collect the reviews.

import requests
from bs4 import BeautifulSoup
from time import sleep

url = "https://www.aliexpress.com/item/3256801798731854.html"

response = requests.get(url).text

soup = BeautifulSoup(response, "html.parser")

reviews = soup.select("div.f-content dl.buyer-review dt.buyer-feedback")

for rev in reviews:
    rev_text = rev.find("span").text
    print(rev_text)
    sleep(1)

The problem is that when I try to run this code, I get nothing in my terminal which is crazy. I really don't understand why my reviews variable is returning an empty list because print(reviews) prints an empty list.

What's wrong with my select statement in Beautifulsoup.

I also don't understand why the code reviews1 = soup.select("div.f-content") doesn't work(prints an empty list) but reviews2 = soup.select("div", class_ = "f-content") works

I have seen this problem with several of my work & I don't understand why reviews1 doesn't work yet it's supposed to be working.

But generally, I would like to kindly get some guidance on my code so that I can be able to collect reviews from any Aliexpress product page.

Solution

Main issue here is that the reviews are rendered in an iframe so your url is another.

Check also this answer, that deals with a selenium solution.

Example

import requests
from bs4 import BeautifulSoup
from time import sleep

url = "https://feedback.aliexpress.com/display/productEvaluation.htm?v=2&productId=1005001985046606&ownerMemberId=227141890&companyId=236776222&memberType=seller&startValidDate=&i18n=true"
response = requests.get(url).text
soup = BeautifulSoup(response)

reviews = soup.select("div.f-content dl.buyer-review dt.buyer-feedback")

for rev in reviews:
    rev_text = rev.find("span").text
    print(rev_text)
    sleep(1)

Answered By - HedgeHog